JP2003023413A - System decoder and method for correcting packet data - Google Patents

System decoder and method for correcting packet data

Info

Publication number
JP2003023413A
JP2003023413A JP2001208680A JP2001208680A JP2003023413A JP 2003023413 A JP2003023413 A JP 2003023413A JP 2001208680 A JP2001208680 A JP 2001208680A JP 2001208680 A JP2001208680 A JP 2001208680A JP 2003023413 A JP2003023413 A JP 2003023413A
Authority
JP
Japan
Prior art keywords
data
packet
decoder
error
video
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP2001208680A
Other languages
Japanese (ja)
Other versions
JP2003023413A5 (en
JP3931595B2 (en
Inventor
Junichi Kimura
Yoshinori Suzuki
Toru Yokoyama
淳一 木村
徹 横山
芳典 鈴木
Original Assignee
Hitachi Ltd
株式会社日立製作所
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hitachi Ltd, 株式会社日立製作所 filed Critical Hitachi Ltd
Priority to JP2001208680A priority Critical patent/JP3931595B2/en
Publication of JP2003023413A5 publication Critical patent/JP2003023413A5/ja
Publication of JP2003023413A publication Critical patent/JP2003023413A/en
Application granted granted Critical
Publication of JP3931595B2 publication Critical patent/JP3931595B2/en
Application status is Active legal-status Critical
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/0078Avoidance of errors by organising the transmitted data in a format specifically designed to deal with errors, e.g. location
    • H04L1/0079Formats for control data
    • H04L1/0082Formats for control data fields explicitly indicating existence of error in data being transmitted, e.g. so that downstream stations can avoid decoding erroneous packet; relays
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/004Arrangements for detecting or preventing errors in the information received by using forward error control
    • H04L1/0045Arrangements at the receiver end
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/12Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/18Automatic repetition systems, e.g. van Duuren system ; ARQ protocols
    • H04L1/1829Arrangements specific to the receiver end
    • H04L1/1835Buffer management
    • H04L1/1838Buffer management for semi-reliable protocols, e.g. for less sensitive applications such as streaming video
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/61Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/89Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41Structure of client; Structure of client peripherals
    • H04N21/414Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
    • H04N21/41407Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network, synchronizing decoder's clock; Client middleware
    • H04N21/438Interfacing the downstream path of the transmission network originating from a server, e.g. retrieving MPEG packets from an IP network
    • H04N21/4382Demodulation or channel decoding, e.g. QPSK demodulation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/61Network physical structure; Signal processing
    • H04N21/6106Network physical structure; Signal processing specially adapted to the downstream path of the transmission network
    • H04N21/6131Network physical structure; Signal processing specially adapted to the downstream path of the transmission network involving transmission via a mobile phone network
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/643Communication protocols
    • H04N21/64322IP
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/12Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals

Abstract

(57) [Summary] [PROBLEMS] When packet loss occurs during wireless communication transmission of digital media data, media data passed to an application decoder has a format in which the payload of the packet in which the loss has occurred is omitted, and transmission error occurs. Can not detect the exact position of. SOLUTION: A system decoder embeds, in a packet loss portion of received data, a data string which is not included in the specification / standard of an application decoder which can explicitly determine that a communication error has occurred, by a system decoder. , It is possible to detect the exact position of the transmission error. Further, by determining the data string to be embedded between the system decoder and the application decoder in advance, more accurate error position detection becomes possible.

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention
Data coding technology for wireless communications.
Data processing, especially when packet loss occurs
It is about. [0002] As a next-generation wireless communication service, data
Digital media compressed and stored on the distribution server
Wireless data terminal such as a mobile phone
It is assumed that the data is received and played back by. like this
Packet loss (packets being transmitted)
The form of communication when data loss occurs
The following two points are assumed. 1) Lost packets (data units, data chunks)
Connectionless type
Transport layer protocols (e.g. UDP: User Datagram Pr
otocol, RTP: Realtime Transport Protocol)
2) Connection that enables retransmission of lost packets
Type of transport layer protocol (e.g., TCP: Transm
ission Control Protocol for all messages
Reliability is required for error detection and response confirmation processing.
Suitable for the application you want. However, the transmission path
If the condition is poor, frequent acknowledgments and retransmission requests
Receive all the data.
Download and start the playback process after storing once
De. (Details of UDP, RTP and TCP will be described later.)
The transport layer is a network that enables heterogeneous communication.
Standard reference model for network architectures (OSI: Open Systems
Interconnection, open system interconnection, communication line
Physical layer-data link layer-network from close to
Layer-transport layer).
Level that plays a role. "Address tube
IP (InternetProto
col) corresponds to the network layer. The above two services
Of the forms, download involves all data loss
Received data to make a retransmission request
Does not include transmission errors. Meanwhile, Streamy
In the streaming service, there are packets with data loss.
Data is not resent even if the
Could be Here, a specific case will be described. Figure 2 is mobile
FIG. 3 is a diagram showing a system configuration of a telephone, and FIG.
This is a representation of the packet to be transmitted. This mobile phone terminal 20
Towards 0, the transport layer packet sequence shown in FIG.
Consider a case where (one type of media data) is transmitted.
One packet in the transport layer regardless of TCP or UDP-RTP
Are header parts of the transport layer (411, 421, 4
31, 441, 451) and the payload portion (412, 42).
2, 432, 442, 452). This
In the example, the payload part is the actual media data.
You. In the header part, the data amount of the payload part (byte unit)
), Packet serial number (sequence number), payroll
Type (only for RTP, multiple types at once for TCP)
It does not deal with data. Video data, audio data
Data is identified using this data) and time stamp
(Only for RTP). First, Ante
The radio signal received from the
The signal is electrically converted by the unit 202 and the data link layer
System decoder 205 in the form of a col packet 203
Is input to The system decoder 205
Data link layer protocol packets
Layer packets, then transport layer packets.
It is protocol-converted and is the payload part of the transport layer
The injury is output to the memory 207. Also, system decoding
Error detection processing in packets, and
Identification of payload type (only for RTP)
U. In addition, media data of different payload types
If they are distributed at the same time,
Configure the media data and synchronize with each other
Output to the memory 207. [0004] The transport layer protocol is TCP.
The system decoder has reached without error
For each packet, the response confirmation information 204
Reply to the face unit 202. These response confirmation information
Is converted to a wireless signal by the wireless interface unit 202.
Is returned to the sender. On the transmitting side, the transmitted packet
Waits for the response of the packet before sending the next packet.
(It is possible to process multiple packets in parallel)
When the response confirmation does not arrive even after a certain period of time
, The packet is retransmitted to the mobile phone terminal 200.
For this reason, no packet loss occurs in TCP, and 2 in FIG.
060, which consists of only the payload
Output to the memory 207 as payload data 206
Is done. The payload data referred to here is generally
In the case of download, a file containing images and sounds
File format, specifically, MPEG-4
MP4 file format, Apple's Quick Tim
e, Microsoft's WMT (Windows Media Technology), etc.
There is. [0005] The transport layer protocol is UDP-RTP
If so, the system decoder checks the response.
No packet loss is allowed. For example, in FIG.
The second packet (421 and 422) arrives at the system decoder
If not, or if the second packet contains a transmission error.
If it is, it will be as shown at 2061 in FIG.
Data consisting of only the payload portion excluding the second packet
Is output to the memory 207 as payload data 206.
It is. The payload data referred to here is streaming
In general, in the case of video, packets of video encoded data are
Data mixed with audio encoded data packets
It becomes. Thus, a plurality of media data (for example,
Video data and audio data)
If you need to play back
The header 205 of the RTP packet contains
The associated media type depends on the payload type
Audio decoder (e.g., video decoder, audio
(Coder) type is decoded and different types of media data
Data to the memory 207 as data. afterwards,
The data stored in the memory 207 is used for streaming.
If the last packet arrives, media
To the application decoder 209 as data 208
Is output. In this specification, application data
Coder 209 is converted to file format file
Video and audio decoders
A generic term for any media decoder and application data
When it is indicated as a coder, the internal configuration is not limited
To In the application decoder, lost packets
While performing the error concealment process of
The media data is restored / reproduced. However, media
Error concealment processing included in the decoder is not standardized
Therefore, the packet loss is included in the media data 208.
Image quality and audio quality of the reproduced image in
It depends on the ability. [0006] Packet loss during transmission
Is passed to the application decoder when
The media data is the payload of the lost packet.
Is omitted. Generally application deco
Data that is not included in the applicable standard decoding method.
Transmission error when a combination is found in the input data
It is determined that it has occurred. Therefore, lost packets
Data combining packets before and after
If the combination is included in the method, transmission error
Can not detect the exact position of. [0007] In addition, data including packet loss can be obtained with high image quality.
Application decoder to transmit
Detects errors and controls so that the effects of errors do not appear in the image quality.
Need to control. Such error correction processing is performed by a decoder.
Product price and power consumption in order to increase
In applications where fast processors are not available
Satisfy the requirements in terms of playback processing speed and image size.
I will not let you. [0008] A packet log of received data is provided.
Application decoder explicitly communicates communication
A data string that can be determined to have an error
Embed in da. Specifically, application decoding
Data strings that are not included in the data specifications
Use as This allows application decoding
Can detect the exact location of transmission errors.
You. Also, the data string to be embedded is updated with the system decoder.
Between application decoders
As a result, more accurate error position detection becomes possible. In addition,
Error using embedded data for error detection
Media data to application decoder specifications
By converting to standard-compliant data,
To reduce the error correction processing load on the
It becomes possible. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First, FIG. 2, FIG. 38, FIG.
6 and FIG. 7, the process for detecting the occurrence of packet loss is performed.
Think of a pair. FIG. 6 shows the system decoder 2 shown in FIG.
FIG. 5 shows the internal configuration of FIG. Entered packet
First, the packet data is transformed by the packet processing unit 301.
Converted to port layer packets. Transport layer
If the protocol is UDP-RTP, the header of each UDP packet
Analyze the error detection information contained in the
If the packet contains an error, the packet
At 301, the packet is discarded. RTP packet is U
In the payload part of the DP packet, the payload data
And sequence number and payload type information.
The RTP header. So send later
Even if the received packet arrives at the receiver first.
Transmitted by the media data reconstruction unit 308 of the system decoder
It is possible to change the order. Therefore,
If the received UDP packet contains no error, R
The TP packet is analyzed, and the payload data 307 and the
Stamp / sequence number / payload type 801
Is output to the media data reconstruction unit 308. On this occasion,
Timestamp, sequence number, payload type 8
01 is also output to the packet loss detection unit 802. to this
Thus, the packet loss detection unit 802 uses the sequence number
And has not arrived by parsing the payload type
Packet data and packet data with transmission errors
Data can be detected for each payload type.
Specifically, the media data reconstructing unit 308 outputs
Sequence number and payroll
The packet type has not reached the packet loss detection unit 802.
The packet is determined to be a lost packet. In addition, pay
Load data 307 and time stamp / sequence number
No./payload type 801 received media data
In the data reconstruction unit 308, the sequence number of each RTP packet is
Per payload type, according to issue and payload type
Perform the operation of changing the order of the packet sequence. For example, figure
As shown in 7 (a), “2 in a certain payload type
The second packet (421 and 422) has not arrived "
Depending on the status of the network used for
Type packets are the first packet, the fourth packet,
Arrived at the portable terminal side in the order of the fifth packet and the third packet
Suppose ". In this case, the packet processing unit 30
1 indicates the arrival packets in the order of arrival (411, 412, 4
41 and 442, 451 and 452, 431 and 432)
412, 442, 45 as shown in FIG.
To the RTP and the payload data string 307
Timestamp, sequence number, payload
Type column 801 (only sequence numbers are displayed in the figure
The media data restructuring unit 308
And output. In the media data reconstruction unit 308,
RTP packet sequence number and payload type
The order of the packet sequence for each payload type according to the
Perform the swap operation. In the example of FIG. 7, as shown in FIG.
Payload data 2061 is transmitted to the system decoder and
Media that acts as an interface with the application
Output to the memory 207 through the
It is. When the media data reconstructing unit 308
Timing that outputs each payload data to the output unit 310
Is typically a timestamp included in each RTP packet
Managed by the application decoder
It is also possible to control while considering the raw situation. FIG.
FIG. 8 is a view showing another embodiment of the mobile phone terminal.
Media data (video data and audio data) is RTP
Shows the case of streaming distribution by
You. In FIG. 38, the mobile phone terminal 200 shown in FIG.
Data from system decoder to application decoder
Data flow. In this case, the RTP packet
The payload type of the header part is video data or
This shows audio data. System decoder 20
51 is described in the header part of each received RTP packet.
Examine the payload type and sequence number
Audio data and audio data.
Payload data 2061 in memory 2071
Output to Output timing of each payload at this time
Is based on the time stamp in the RTP packet header,
Video and audio are synchronized with each other during playback
Can be decided. Each media stored in the memory 2071
Data 2081 reaches the data amount of the reproduction unit.
At that point, the video in the application decoder 2091 is
Input to the video decoder and audio decoder,
Playback processing is performed. On the other hand, the protocol of the transport layer is TC
If it is P, it is included in the header of each TCP packet.
The error detection information is analyzed and the error detection result 302 is
The information is passed to retransmission determination section 304. At this time, the received packet
If the packet includes the packet, the packet is discarded. Again
In the transmission determination unit 304, the error detection result 302
If "None" is indicated, the response confirmation information 305
The answer confirmation unit 306 is notified. This response confirmation information 305
Is converted into the wireless signal 204 by the response confirmation unit 306,
Notifying the transmitting side through the wireless interface unit 202
You. On the sending side, the response is not
Retransmits unacknowledged packets to the mobile phone terminal 200
You. If the protocol is TCP, the sender's packet
The transmission is performed in consideration of the arrival status of the response confirmation,
Basically, no illegal packets are generated. However,
Recently, download reliability and streaming
A service with performance that is in the middle of real-time
Thus, pseudo-streaming has also been proposed. This pseudo-s
Trimming is "as reliable as downloading
Use connection-oriented transport layer protocol
However, the media data is divided into several files and sent.
Without waiting for the reception of all the data necessary for playback.
Play from the media data in the received file as appropriate.
”. Download the communication reliability
Is high, but the playback process continues until the file is completely received.
Can not start. So like a mobile phone
For devices that cannot incorporate large memory into
Media data cannot be received. So
Here, the media data is divided in advance on the transmitting side.
Processing by performing playback processing from the arrived file.
To delete files that have been deleted from memory
The law becomes effective. However, pseudo-storage by wireless communication
When performing reaming, retransmission of packets frequently occurs.
Then, even if the playback of one file ends, the next file
File has not arrived and playback has to be paused
Problem occurs. So, with pseudo streaming
Is a communication method that has a connection with the sender, such as TCP
Even if it is a formula, a specification that allows some packet error
The necessity of incorporating a combination arises. Note that this packet
The method to allow the error is to delay from the user command.
If you think about it, it will be effective even if you download it. [0011] When the packet loss is allowed by TCP,
The packet loss detection method is a method that allows packet loss.
belongs to. To allow for packet loss,
Alternatively, the response confirmation procedure can be controlled on either the terminal side.
No. To control on the sending side, respond after a certain period of time
If the confirmation does not arrive or if the number of retransmissions exceeds a certain number
For example, by giving up acknowledgment under conditions such as
Will be revealed. In this case, the occurrence of packet loss
Can be confirmed in the sequence number string of the
Indicates to the terminal the number of packets to be acknowledged at the same time.
The sequence number is larger than the condition.
Packet loss can be determined when a packet arrives.
). However, in this method, the application on the terminal side
The progress of the playback process in the decoder
Processing of each file because it cannot be reflected in control
If there is a gap between the estimated time and the actual value,
There may be situations where processing must be suspended.
You. Therefore, the method of controlling the response confirmation processing on the terminal side next
think about. Specifically, retransmission determination section 304 in FIG.
And a packet loss detection unit 802. Packet processing
The unit 301 is included in the header part of each TCP packet
Analyzes the error detection information and resends the error detection result 302
The information is passed to the determination unit 304. At this time, the received packet
If so, the packet is discarded. on the other hand,
If the received packet contains no errors, the TCP
The packet is analyzed and the payload data 307 and the sequence
Port number 801 (there is no payload type in TCP
) To the media data reconstruction unit 308. simultaneous
To the packet loss detection unit 802
Output. Thereby, the packet loss detection unit 802
Is not received by analyzing the sequence number
Packets can be detected. Specifically,
Sequence number has not arrived from the packet processing unit 301
Judge the packet as a packet that has not been received
You. In the packet loss detection unit 802, packets before and after
After receiving, the specified time (controlled by the system decoder)
A packet that is not received even after elapse is regarded as a lost packet,
The sequence number 807 is notified to the retransmission determination unit 304.
You. In the retransmission determination unit 304, the retransmission of the lost packet
It is determined whether or not there is a problem in the reproduction process. And resend
If you decide that there is a problem with the playback processing delay if you wait
Notifies the response confirmation information 305 to the response confirmation unit 306
Together with the retransmission decision result indicating that retransmission is not performed.
Response number 303 to the packet loss detector 802
I do. It is determined that the playback process will proceed without delay even after waiting for retransmission.
If retransmission is possible, a retransmission determination result indicating that retransmission is to be performed
-The sequence number 303 is sent to the packet loss detection unit 802.
reply. From this response result, the packet loss detection unit 80
2 has been received by the media data reconstruction unit 308
If there are no packets, the packets waiting to be resent
Has been notified).
Check if the packet is acceptable as a
(A retransmission determination result indicating that the
Can be turned off. For packets waiting to be retransmitted,
Even if a fixed time (controlled by the system decoder) elapses,
If not, the retransmission determination section 304
No. 807 is notified. In retransmission determination section 304, the number of retransmissions
The possibility of a retransmission request taking into account
U. The retransmission determination unit 304 has determined that the packet is a lost packet.
Later, process received packets as usual
This makes it possible to rewrite information. like this
Receives payload data 307 and sequence number
In the media data reconstructing unit 308, the TCP
Output received packet according to packet sequence number
An operation of changing the order is performed. After replacement,
The data reconstructing unit 308 is a media data output unit 310
The timing of outputting the payload data to the
Controlled at regular intervals,
It is also possible to control while considering the playback status of the
You. FIG. 39 is a diagram showing a further embodiment of the mobile phone terminal.
With two types of media data (video data and audio data)
MP4 file that integrates and accumulates
Format data is distributed by pseudo-streaming using TCP.
It shows the case when it is received. In FIG. 39, FIG.
The mobile phone 200 shown in FIG.
The data flow to the application decoder is different.
You. The system decoder 2052 is a TCP packet header part
Investigate the sequence numbers of
The load data 2062 is output to the memory 2072. this
The output timing of each payload at the time
Considering the processing status of media data in the decoder
Decide. In addition, the loss packet in the system decoder is
Processing status of application decoder for retransmission control
Reflect the situation. The media data stored in the memory 2072
Data 2082 is one file format data and
When the application decoder 209 is established,
2 is input to the file decoder. File decoding
In the header, the information in the file header
Media data (here, video data and audio data
Data is included), and
Input to the video decoder and audio decoder
You. FIG. 40 shows another embodiment of the mobile phone terminal, and FIG.
Another embodiment of the stem decoder is shown in the application
Considering the processing situation in the decoder,
Of lost packet and error packet retransmission request
The flow of data for performing is described. As shown in FIG.
Data 210 and 211 correspond to processing status related information
You. As the data 211, the unprocessed data in the memory
The remaining amount and the data 210 include the state of occurrence of reproduction delay,
The frame rate and the bit rate are included. This
These data are sent to retransmission determination section 304 in FIG.
You. The retransmission determination unit 304 obtains the
Data remaining and the amount of data
Information about the cost rate, and also when the retransmission packet arrives
Estimated time (based on the data arrival status up to that point)
Measurement), the importance of the packet (e.g., file
Format data header and video data sequence
Losing the header makes it difficult to reproduce the entire data.)
Waiting for packet retransmission to the application decoder
The impact of Method of controlling packet loss in TCP
In addition to the above, both the transmitting side and the receiving terminal side
It is also possible to control. In this case, the response confirmation information
In addition, a mechanism to send a retransmission stop request or retransmission request from the terminal side
By making the only ones, more precise and faster control becomes possible. A pseudo-stream using TCP described above
Ming, only one type of media data can be transmitted in one communication
There is a problem that it cannot be handled. Therefore, media data
Each file divided into must have a file header
This may increase the communication capacity. There
In pseudo streaming, a communication method called TCP-RTP
Is also effective. Here, the response confirmation processing is controlled on the terminal side.
The mechanism of TCP-RTP is explained in Fig.6.
You. The packet processing unit 301 includes a header for each TCP packet.
Analyzes the error detection information contained in the section and detects errors
The result 302 is passed to the retransmission determination unit 304. At this time,
If the packet contains an error,
Discard. On the other hand, the received packet contains no errors
If included in the TCP packet header and TCP payload
RTP payload data 307 and
TCP sequence number, RTP time stamp, RTP sequence
Media number with RTP payload type 801
Output to the reconstruction unit 308. TCP sequence number at the same time
・ RTP time stamp ・ RTP sequence number ・ RTP payroll
Output the packet type 801 to the packet loss detection unit 802.
You. Accordingly, the packet loss detection unit 802 uses the TCP
Sequence number, RTP payload type, RTP sequence
By analyzing the number, it
The payload type can be detected. Ingredient
Specifically, the TCP sequence number is sent from the packet processing unit 301.
Packets that have not arrived, packets that have not been received
RTP payload type and RTP sequence
The type is determined from the number. Packet loss detector 80
In 2, the specified time after the previous and next TCP packet is received
Not received even after (controlled by system decoder) has elapsed
A TCP packet is regarded as a lost packet, and its TCP sequence number is
No. 807 is notified to the retransmission determination unit 304. Retransmission determination unit 3
In 04, retransmission of the lost packet hinders the playback process.
It is determined whether or not there is. And waiting for retransmission delays playback processing.
If you determine that there is a problem in terms of delay,
305 to the response confirmation unit 306 and resend
Retransmission determination result indicating not to perform TCP sequence number
303 is returned to the packet loss detection unit 802. Resend
When it can be determined that the playback process proceeds without delay even after waiting
Contains the retransmission judgment result indicating retransmission
Response number 303 to the packet loss detection unit 802.
You. From this response result, the packet loss detection unit 802
TC not received by media data reconstruction unit 308
If the P packet is a packet waiting to be retransmitted (indicating that
Retransmission judgment result has been notified) or
Check if the packet is acceptable as a loss (do not retransmit
Has been notified of the retransmission decision result indicating that the
Wear. For packets waiting to be retransmitted,
Is not received even if the time (controlled by the system decoder) elapses
In this case, the TCP sequence number is
No. 807 is notified. In retransmission determination section 304, the number of retransmissions
The possibility of a retransmission request taking into account
U. The retransmission determination unit 304 has determined that the packet is a lost packet.
Later, process received packets as usual
This makes it possible to rewrite information. like this
In the procedure, payload data 307 and TCP sequence number
・ RTP time stamp ・ RTP sequence number ・ RTP payroll
Media data reconstruction unit 308 that has received the
Now, the sequence number of each RTP packet and the payload
Packet sequence for each payload type according to the load type
Is performed. Media data re-
The configuration unit 308 sends the media data output unit 310
For the timing of outputting the load data, refer to each RTP
The time stamp and application data included in the
It is effective to control while considering the playback status of the coder.
is there. Of each payload output from the system decoder
The reproduction method is described, for example, with reference to FIG. Two kinds
Media data (video data and audio data) is TC
In the case of pseudo-streaming distribution by P-RTP, RTP
The payload type of the packet header is video data
Alternatively, it indicates audio data. System deco
The header 2051 is recorded in the header of each received RTP packet.
Investigate payload type and sequence number listed
Video data and audio data.
Memory payload data 2061 in sequence number order
2071. The output tie of each payload at this time
The time stamp and the time stamp in the RTP packet header.
Processing of media data in application decoder
It is decided in consideration of the situation. Also, the system decoder
Application data for retransmission control of lost packets
Reflects the processing status of the coder. System decoder
Packet retransmission control in the same way as TCP
40 and FIG. 41. The data shown in FIG.
Data 210 and 211 correspond to the processing status related information.
You. The data 211 includes each payload tag in the memory.
Remaining amount of unprocessed data in IP, as data 210
The playback delay, frame rate and bit
Trates and the like are included. These data are shown in FIG.
Sent to retransmission determination section 304. In retransmission determination section 304,
The data remaining amount obtained by the data 211 and the data 21
Information about the consumption rate of data obtained from 0,
An estimate of the time it takes for a retransmitted packet to arrive (
Inferred from the data arrival status at
The importance of audio
No), lost packet payload type percentage,
From the payload type time stamp, etc., the packet
Estimate the effect of waiting on the application decoder
I do. Then, each media data stored in the memory 207 is stored.
Data 208 is “TCP communication in pseudo streaming.
Waiting from reception to start of playback considering transmission speed
When the amount of data reaches the specified
Video decoder in the application decoder 209
And input to the audio decoder for playback processing.
You. Here, the operation of the packet processing unit 301 will be described.
And touch it. FIG. 8 is a block diagram of a wireless terminal of IS-95 standard.
3 shows details of a packet processing unit. First, the wireless
The data link layer system is used by the wireless interface unit 202
RLP (Radio Ling Protocol)
Frame data 902 (both frames and packets are data
Unit) and is input to the RLP analysis unit. FIG.
1 shows the data format of the RLP frame. RLP is CD
A communication protocol that supports MA data.
To maintain the interface with the client layer protocol
You. In the RLP analysis unit 903, based on the RLP header information,
Combining multiple RLP payload data 904
With a frame compatible with PPP (Point to Point Protocol).
And constructs the data 9041 and outputs it to the PPP analysis unit 905.
You. Fig. 10 shows the data format of the PPP frame.
You. PPP is a type of data link layer protocol
Used on serial lines. The data link layer is mainly
The role of "establishing physical communication" and "
Detect errors that occur while passing through "role
Fulfill. In the PPP analysis unit 905, the network used
Work layer protocol determination (dissolve upper layer protocol data
Analysis), errors that occur while data is
Detection (analysis of error check data) and multiple pay
IP packet data 9 combining load data 906
The processing of the configuration 061 is performed. Note that here,
IPv4 is the protocol of the network layer that is the protocol
List as adopted (commonly known
Other IPs include IPv6, AppleTalk, etc.
The present invention is applicable to the present invention). Figure 9 shows the data of an IPv4 packet.
Indicates the data format. IPv4 packet data 906
Is input to the IP analysis unit 907. In the IP analysis part, use
Determine which transport layer protocol is used (upper protocol
Protocol data) and header error detection (check
Analysis of the checksum data) and multiple IP
By combining the load data 908,
Construct a packet of cols. And the upper packet is TC
If it is P, the TCP analysis unit 910 sends the TCP packet 9
080 is output, and if it is UDP, the UDP analysis unit 91
1 to output a UDP packet 9081. Figure 12 shows the TCP packet
The data format of the UDP packet is shown in FIG.
Indicates the data format. Transmission between TCP and UDP
This depends on the nature of the data, and is generally less than the transmission speed.
For data where reliability is important, TCP and real-time
UDP is used when necessary. In the TCP analysis unit 910,
Analyzes the checksum included in the header of the TCP packet
By doing so, the error detection processing is performed, and the error detection result 3
02 is passed to retransmission determination section 304 (error shown in FIG. 6).
The same as the detection result 302 and the retransmission determination unit 304).
If the error detection result indicates no error,
The load data 307 is the media data reconstruction unit 308
The source sequence number 801 is
2 and passed to the media data reconstruction unit 308. on the other hand,
The UDP analysis unit converts the UDP payload data into RTP packets.
9121 is output to the RTP analysis unit 913. UDP packet
Checksum is an optional feature that detects errors
Is not essential, but provides a stable communication
It is recommended that they be used. RTP analyzer 913
Then, the payload 307 is transferred to the media data reconstructing unit 30.
8, time stamp, sequence number, and payload
Type 801 and the media data
Assigned to the configuration unit 308. FIG. 42 shows the processing of the TCP unit of FIG. 8 changed to TCP-RTP.
FIG. The difference from Fig. 8 is that the TCP-RTP
Minutes. The TCP analysis unit includes
Error detection processing by analyzing the checksum
Performed, and the error detection result 302 is passed to the retransmission determination unit 304.
(The error detection result 302 shown in FIG.
(Same as the transmission determination unit 304). Error detection result is no error
Indicates that the TCP payload data is
To be passed to the RTP analysis unit as erase data 9120
The TCP sequence number 9122 in the TCP header is
Pass to the Toros section and media data reconstruction section. RTP analyzer
At 913, the RTP payload 307 is re-
The time stamp and sequence of the RTP header are
Share the payload number and payload type with the TCP sequence number.
The packet detector 802 and the media data reconstructor 30
Distribute to 8. Next, the packet loss detector 802 in FIG.
Using the obtained packet loss information, the packet loss position
To notify application decoder explicitly
Think. As shown in FIG. 5, packet loss occurs.
In this case, the data of the lost packet part is omitted.
Will be passed to the application decoder.
Information on kettle loss is notified to the application decoder
Not done. Therefore, the application decoder
Considering the occurrence of
The application decoder is correct.
It will be left to the detection capability. For example, lost
The data string when connecting the front and rear of the packet is accidentally
If included in the application decoder specification,
It may be a cause of delay in detection of ket loss.
You. In the present invention, the application decoder
In order to be able to accurately determine the location of the loss,
Application decoder understands packet loss part
Consider a mechanism for embedding an unencoded coded sequence. Specifically
Allows for error detection in the application decoder.
For simplicity, data strings (e.g.
Error detection embedded data) in the packet loss area
Put in. Data that the application decoder cannot understand
In order to determine the column as an error,
It can be detected almost exactly. This allows the decoder to
Error that causes error data to be treated as normal data
ー Detection delay is avoided, and reproduced image and audio are suppressed
it can. In addition, the application decoder
File format file decoder and video
Audio decoder, audio decoder, scene description decoder
Media processing such as (display position playback processing of multiple objects)
An decoder. Therefore, application deco
Data strings that cannot be understood by the
Data) is the decoding included in the application decoder.
It depends on the type of da. For example, as shown in FIG.
Application decoder decodes MP4 files
Video and audio media decoder
File decoder, video decoder,
A data string that all audio decoders cannot understand
"A coded sequence that the application decoder cannot understand
(Embedded data for error detection) ''
If the packet corresponds to the header of an MP4 file,
Code strings and video data that only file decoders cannot understand
Only the video decoder cannot understand
Code string or audio data section
Embedding a code string that only a video decoder can understand
Not understandable as a whole application decoder
Code strings (embedded data for error detection) may be defined.
No. In addition, as shown in FIG.
When composed of audio decoder and video decoder
In the system decoder, the received packet is
Data and audio data are processed separately.
Embedded data for error detection that differs depending on the load type
Prepare the application decoder as a whole and understand
Code string (embedded data for error detection)
Just fine. The procedure for embedding the data string for error detection is as follows.
It is as described in. As mentioned earlier, media
Of packets that have not reached the data reconstruction unit 308
Kens number (and payload type) and retransmission status
(Retransmission processing / packet loss) indicates that the packet loss detection unit 8
02. Therefore, FIG. 6 and FIG.
As shown in FIG.
Created by the loss detection unit 802 and a packet loss has occurred
Shared with sequence number (and payload type) 806
Notification mechanism to the media decoder reconstruction unit 308
Provided in the system decoder. And Media Deco
The reconfiguration unit 308 determines the sequence number of the lost packet.
Number (and payload type) 806
Then, the embedded data for error detection 805 is output. This
This makes it possible to embed error detection data.
You. Here, packet loss has occurred for simplicity.
Application deco that decodes part of media data
If the decoder is based on the MPEG-4 video standard,
And the payload type of the RTP packet was video
As an example, consider the type of embedded data. Figure
15 is a case where the second packet is lost as shown in FIG.
It is an example of error detection embedded data in the case. MPEG-4 video
In the video standard, 24 or more zeros must be continued in binary notation
Is forbidden. Therefore, the embedded data 8052
Is a data string not included in the MPEG-4 video standard,
The video decoder did not understand and recognized that an error occurred.
Understandable. However, as in this example, the application
With a video decoder (here, a video decoder).
Embed unique words with no rules for error detection
When used as only data, the packet loss position
Although it is possible to detect errors in the surrounding area,
It is difficult to accurately detect the gap between data and normal data.
No. Therefore, existing commercial video decoder is used as it is
The packet log when developing a new system decoder.
Is effective as a position detection method, but has sufficient detection performance.
Cannot be obtained. Therefore, FIG.
Unique word 8051 which we decided beforehand with da
Here is an example of the MPEG-4 video standard when embedded. MP
In the EG-4 video, the combination of 23 0s and 1 1
This is an identification code to indicate a break in data.
2 bytes of data following the 24 bits of
It is designed to show the kind. Used in Figure 1,
Hexadecimal numbers A1 and A5 are the identification codes for the reserve.
And is not used in current standards. Subordinate
This kind of unique word is the standard for MPEG-4 video.
The decoder does not understand and will be considered an error.
You. Therefore, such a data string is stored in advance for error detection.
That the data is embedded data
By understanding the loss detector 802,
Accurate packet loss position detection with video decoder
Of normal data and embedded data for error detection
The eyes are also clear. Also set two unique words
And replace them with the first and last embedded data for error detection.
It is also effective to attach it to In this method, the first
Accidental decoding of a few bits of a unique word
Even if an error is detected in the middle of the embedded data section
If you work it out (between 24 or more 0),
Normal data by searching for the second unique word
Can be detected accurately. Ma
Also, once processed data can be searched again
If so, go back a little bit and then the first unique word
By searching again, the position where packet loss starts
Position can be detected accurately. FIG. 16 shows, in addition to the media data,
This is a method of preparing error notification data of the same size as the above. Mistake
Each binary data in the notification data
One-to-one correspondence with each binary data in the data
In advance, for example, the embedded data 8 in the media data
1 is assigned to the section of 053, and to the section of the normal packet
0 is assigned in advance. The error notification data
Application decoder correctly detects the occurrence of packet loss.
Understandable. At this time, the embedded data 8053 must be
Need not be a unique word. However, error notification data
Must be configured to pass data to the application decoder.
It becomes important. For example, as shown in FIG.
Error notification data packet 706 generated by output unit 802
Is output from the error notification output unit 804 to the memory 207.
It works. In this case, the overall configuration of the mobile phone terminal is
It looks like 17. The difference between FIG. 2 and FIG.
Data packet 706 is sent from system decoder 205
The point passed to the memory 207 and the memory
Error notification data 703 is synchronized with the media data.
Is output to the application decoder 209.
You. FIGS. 43 and 44 correspond to FIGS. 38 and
The function of error notification data is added to the mobile phone terminal configuration in FIG.
This is shown. In FIG. 43 using RTP,
Error notification data packet 7061 and
Error notification data 7031 is processed. Against this
TCP is used, and TCP packets are
In FIG. 44, which does not have a
Error notification data is sent to each media
Distributed to the decoder. In FIG. 44, the file
Packet loss occurs in the format data header
The file header, the information in the header section is
Information needs to be corrected. However, the playback process is more reliable
Is performed, the retransmission determination unit 30 in FIG. 6 or FIG.
In 4, the lost packet is generated in the file header.
Request a retransmission until the data arrives,
It is better to decide the file structure between the sender and the terminal
No. Up to this point, packets have been transmitted to the media decoder.
We've talked about how to report a loss.
The quality of the reproduced image and sound
It depends on the performance of the application decoder. That
The application decoder itself is an advanced error correction
If there is no step, embedded data for error detection
Data may adversely affect decoding performance.
You. Also, application decoder has advanced error correction
In order to have the means, it is necessary to perform processing such as error detection and error recovery.
Must be added to the application decoder,
Also increase. Therefore, memory requirements, CPU power and
Real-time communication terminals with demanding power consumption etc.
To realize reproduction, only simple error correction means can be adopted.
It may not be possible. Therefore, in this application, the file
For transfer type services, embedded data for error detection
Before being processed by the application decoder using the data
Media data with packet loss
Modify data to conform to specifications such as audio and standards
A method is also proposed. The portable telephone terminal 2003 shown in FIG.
Modification of media data to 17 mobile phone terminals 2002
The processing configuration is added. Media to memory 207
Processing after data input will be described below. Error
In the period marker detection 701, the media processing standard
The specified identification code (synchronization marker) for securing synchronization
Error detection padding embedded by the system decoder 205
The embedded data 805 is searched in the memory 207.
Of the media data 208 (see FIG. 16).
Error notification data is used,
Data 703 is also searched in parallel, and errors in media data are searched.
ー Instead of the embedded data for detection,
Data string indicating the error occurrence position described in
Do). Then, when any of them is detected,
The result is a synchronization marker and error detection embedded data (error
If notification data is used, the error notification data
(Data string indicating the error occurrence position described in
Data correction processing is performed on the identification information 713 indicating whether there was a shift.
Notify section 707. At this time, the switch 705
Control information 704. Control information 704 is
Data from the start position to the detection position is stored in the second memory 710.
Indicates to which of the data correction processing units 707 the data should be output
If the detection result is a synchronization marker in the information, the second memory 7
10, the detection result is embedded data for error detection (error
If notification data is used, the error notification data
(A data string indicating the error occurrence position described in
Indicates that data is output to the data processing unit 707.
You. The switch 705 switches according to the value of the control information 704.
Switch from the detection start position to the next synchronization marker
Media data of the second memory 710 or data
The error notification data is passed to the correction processing unit 707.
Is used, and the detection result is
Data string indicating the error occurrence position described in the data
In some cases, error notification data corresponding to the media data
The data 703 is also passed to the data correction processing 707 at the same time).
Thereafter, the memory 207 detects the error / synchronization marker 70
Delete the information up to the data position detected in Step 1
Or, move the pointer to the detected position. Error notification
Data is processed in the same way). In the data correction processing unit 707,
Inspects input data and corrects data to comply with standard system
And outputs it to the second memory 710. At this time, data processing
A switch system is applied from the correction unit 707 to the switch 705.
The control information 712 may be notified (details will be described later). Ma
In addition, the data correction processing unit 707 has already
When you need the data that has been passed to the memory 710
Retrieves the processed data 709 from the second memory and enters
Hand. The second memory 710 responds to a request from a user.
Media data 7 to the application decoder 209
11 is output. In this case, the application decoder
209 does not require advanced error correction processing (error notification
Data is used and the application decoder is incorrect.
If you have a mechanism to receive notification data,
Pass the error notification data to the application decoder
Is also possible). The data correction process described here is a system
The media data is already
If it is divided into audio data,
Correction processing may be performed for each media data of MP4 format.
Multiple media data like one file
If the data is integrated in the file,
Different correction processes need to be performed accordingly. In particular,
Each media data section in the file (video data,
Audio data) and file header
Process. Then, the contents of the correction in the media data section
Therefore, the data capacity etc. described in the file header
Further rewriting is required. Thus, the file header
Data to the file format specification.
Correction allows the file format decoder to
Media data can be assigned to the corresponding decoder
become able to. Next, taking the MPEG-4 video standard as an example,
The processing in the data correction processing unit 707 will be described in detail. Description
First, the decoding method of the MPEG-4 video standard and
The data structure will be described. Moving images handled by MPEG-4
The frame includes one luminance signal (Y signal) as shown in FIG.
No .: 2001) and two color difference signals (Cr signal: 2002, Cb signal: 20)
03), and the image size of the color difference signal is
Both are 1/2 of the luminance signal. In the MPEG-4 video standard, dynamic
Each frame of the image is divided into small blocks as shown in FIG.
Playback process in units of blocks called macroblocks.
Work. FIG. 24 shows the structure of a macroblock. macro
The block is one Y signal block 21 of 16 × 16 pixels.
01 and an 8 × 8 pixel Cr signal block that spatially matches
Lock 2102 and Cb signal block 2103
Has been established. Note that the Y signal block is
In the process of restoring the block, four 8 × 8 pixel blocks (21
011,1012,21013,21014)
May be processed. MPEG-4 video coding algorithm
The algorithm is called MC-DCT (Motion Compensation-Discrete Cosine Transform).
The playback processing is performed in macroblock units as described above.
Makes sense. Motion compensation means that the target
Extract the part similar to the contents of the rock block and its movement
Is a method of encoding the amount
The coded pair of the block area of the previous frame and the original image
Elephant blocks (21011, 1012, 21013, 2
1014, 2102, 2103)
However, after the frequency transform by DCT, each transform coefficient is quantized.
And encoded. Looking more closely, the MPEG-4 video standard
In-frame coding (intra coding), frame
Inter-coding (prediction coding), a code called bidirectional coding
There is a conversion method. Intra coding is the difference after motion compensation
Input block image to encode, not block image
Is directly applied to DCT, and each transform coefficient is quantized and encoded.
Is a data compression method for
On the other hand, a frame to which intra coding is applied is referred to as an I-VOP (Intr
a-coded Video Object Plane.
(Synonymous with the name). I-VOP is the decoding information of the past frame.
Information is not required, so decryption at random access starts
Used as a frame. Predictive coding refers to MC-DCT
The compression method used, especially for temporally past frames
Is an encoding method that performs MC. And the frame in the frame
Prediction coding or intra coding for black blocks
P-VOP (Predictive-coded VOP)
Call. In addition to this, temporal and past frames
There is also a method of performing MC using information (bidirectional coding),
Frames using the encoding method are B-VOP (Bidirectional
y predicted-coded VOP). FIG. 19 is a block diagram of general MPEG-4 video decoding.
It is a lock figure. First, the input media data
Meaning from binary code by decoding unit 501
Is converted to decoded information having And the DCT coefficient quantum
Information related to the quantized data is sent to the inverse quantization unit 502.
After being decoded into DCT coefficient data, the inverse DCT unit 503
The difference macro block image is restored. On the other hand,
Information about the motion compensation is sent to the motion compensation unit 504. Motion compensator
In step 504, the frame storing the data of the previous frame is stored.
From the memory 507 according to the decoded amount of motion
The macroblock image is restored. Restore in this way
Difference macroblock image and predicted macroblock image
The restored macroblock image is generated by the adder 505
Are synthesized by the synthesizing unit 506 into the restored frame image.
You. The restored frame image is output to the display processing unit.
And the frame menu for motion compensation of the next frame.
Stored in the memory 507. FIG. 20 shows error detection and
MPEG-4 video decoding block with error correction processing function
FIG. The decoding unit 5012 decodes the input data.
Error detection is performed together with the analysis. Various types of error detection
Although there is a method, in general, the coding of MPEG-4 video coding
Performed by detecting variable-length codes that do not exist in the book
It is. If a data error is detected, the decoding unit 5
012 outputs the target data to the restoration processing unit 509. Ma
In addition, the decoding unit 5012 determines when the data error is detected.
Switch the output control information 510 of the predicted macroblock image.
Switch 508 and the predicted macroblock image is restored.
Control section 509. Code decoding unit 5012
Error detection processing and processing of the recovery processing unit 509
Content is not standardized and includes transmission errors
The playback quality of data depends on the specifications of each product.
You. FIG. 21, FIG. 22 and FIG.
This shows the basic data structure of video coding.
You. FIG. 21 shows the overall data structure, and FIG. 22 shows the frame header.
FIG. 25 shows the data structure of each macroblock.
The structure is shown. In FIG. 21, the VOS header is MPEG-4
Profile-level information that determines the scope of the video product
Information, VO header determines the data structure of MPEG-4 video encoding
Version information, VOL header is image size,
Information such as data rate, frame memory size, and application tools.
Information. Both decode the received encoded data
This is essential information for GOV header contains time information
Included, but not required and can be omitted
It is. VOS header, VO header, VOL header, GOV header
Both start with a 32-bit unique word,
Searchable. End code of VOS indicating the end of the sequence
Are also 32-bit unique words. These unique
A word starts with 23 0s and 1 1s, and
The two bytes of data following the packet indicate the type of delimiter
It has such a structure. Each frame (MPEG-4
(Called VOP in video). VOP
Starts from the VOP header shown in FIG.
The macroblock data of the structure to be
The structure is continued from left to right. FIG.
It is a data structure of a VOP header. Call VOP start code
Start with a 32-bit unique word. vop_codi
ng_type is the coding type of the VOP (I-VOP, P-VOP, B-VOP, etc.)
, Followed by modulo_time_base and vop_time
_increment is a time stamp indicating the output time of the VOP
It has become. modulo_time_base is information in seconds, vop_
time_increment is information of seconds or less. vop_time_incre
For information on the accuracy of ment, refer to vop_time_increm of the VOL header.
Indicated in ent_resolution. modulo_time_base information
Is the difference between the value of the previous VOP in seconds and the value of the current VOP in seconds.
And 1 is encoded for the number of changes. One
That is, if the time in seconds is the same as the previous VOP, 0 is set to 1 second
If the time differs by 10 minutes, 110 if the time differs by 2 seconds
Encoded. vop_time_increment information for each VOP
Subsecond information in vop_time_increment_resolutio
It is shown with the precision indicated by n. intra_dc_vlc_thr contains
DC component of DCT coefficient in intra-coded macroblock
Is encoded with an encoding table different from the AC component.
Or encoded in the same encoding table
Information is included. This intra_dc_vlc_thr
Value and quantization accuracy of DCT coefficient in each macroblock
From which macro table to use
Selected on a per-block basis. vop_quant quantizes DCT coefficients
This value indicates the quantization accuracy when performing
Initialization accuracy. vop_fcode_forward and vop_fcode
_backward indicates the maximum range of motion in MC
You. FIG. 25 shows the basic data structure of a macroblock (I-VOP
And P-VOP). not_coded is used only in P-VOP 1
A bit flag that specifies
Indicates if data is continued, 0 if
Means that the data for that macroblock follows.
In the case of 1, if the following data is
Lock related data, and
The signal is copied from the same position in the previous frame.
Is generated. mcbpc can be 1 to 9 bits
This is a variable length code, and the encoding type of the macroblock is
Mb_type, and the encoding within each of the two chrominance blocks
Cbpc indicating whether there is a quantized DCT coefficient (a value other than 0)
(In the case of an intra-coded block, the quantized DCT coefficient
(Indicating whether or not there is an AC component) with one code.
You. The coding types indicated by mb_type include intra, intra + q,
inter, inter + q, inter4v (inter4v is the
The unit for performing the compensation is 2101 instead of 2101 in FIG.
Be in units of four small blocks from 11 to 21014
Stuffing), intra and intra + q
Intra coding, inter, inter + q and inter4v
Measurement coding and stuffing are used to adjust the coding rate.
Indicates that the data is mee data. + Q is the DCT coefficient
Quantization accuracy of the previous block (quant) or initial value
(vop_quant, in the first coded macroblock of the frame
Application). Stuffin
In the case of g, the data after ac_pred_flag in FIG.
Omitted, the decoded mcbpc and not_coded values are also
It is not reflected in the synthesis of the raw image. ac_pred_flag is mb_t
Information included only when ype is intra-coded
And the AC component of the DCT coefficient from the surrounding blocks
Indicates whether to make a prediction. If this value is 1, the AC component
Part of the quantized playback value of
Has become. cbpy is a variable length code of 1 to 6 bits and 4 codes
Quantized DCT coefficients coded in each luminance block
(Outside value).
In the case of an intra-coded block, the AC of the quantized DCT coefficient
Component is present or not). For dquant, mb_type is intra +
exists only if q or inter + q
The difference value from the quantization accuracy value of
dquant is the quant of the macroblock. Motion vector
Information about the encoding of the
Only included. Intra difference DC component has mb_type
Tra coding, and use_intra_dc_vlc is 1
It is information that is included only. In MPEG-4 video, intra
The DC component of the DCT coefficient in the coding block
The difference between the DC component of the DCT coefficient and the DC
It is a child. The quantization method is also different from the AC component.
For the encoding method, a method different from the AC component is prepared. However
Then, by setting use_intra_dc_vlc to 0, the amount of DC component
For the quantization value, apply the same encoding method as the quantization value of the AC component.
It becomes possible. The value of use_intra_dc_vlc is
Intra_dc_vlc_thr defined in the VOP header and the
Determined by the quant of the rock block. Intra AC component or I
For the DC / AC component, refer to “cbpy, cbpc
It is indicated that a value other than 0 exists for the quantization coefficient.
Only "block has information. The macro block data shown in FIG.
Data structure, once data synchronization is lost due to transmission errors and the like.
Data synchronization until the start code of the next frame
Cannot be recovered. Therefore, the MPEG-4 video standard
Provides a separate data structure for error resilience. Ingredient
It is called video packet as well as data splitting
Processing is prepared, and its application is specified in the VOL header.
Is determined. Video packets are several macroblocks
Is a data unit that collects each video packet data.
At the beginning contains the synchronization marker and the position information within the frame
A video packet header is arranged. Also, block
The prediction of data between nodes is also closed within the packet. That
Therefore, even if data synchronization is lost, one
Of the next video packet.
It is also possible to restore synchronization at the beginning of the packet. FIG.
Is an example of video packet division. In FIG. 26, 230
5, 302, 2303, 2304, 2305
It is divided into video packets. The first packet of the frame
At the beginning of each packet data except for the packet 2301,
The video packet header shown in FIG.
For the first packet of the system, the VOP header of FIG.
Is placed. The data structure of the packet header is resync_mar
Start with a synchronization marker called ker. This synchronization marker
Is a unique word of 17 to 23 bits and its data
The size is the frame encoding type and the amount of motion during motion compensation
Is uniquely determined from the range. macroblock_numbe
r is the position in the frame of the first macroblock in the packet
It is shown. With the first two data,
The period recovers completely. quant_scale is the video packet
This is the initial value of the quantization accuracy of the DCT coefficient in the unit. This
Is the first coded macro block of the video packet.
Applied to the previous macroblock quant
You. header_extension_code is v from modulo_time_base
The data up to op_fcode_backward is
1-bit flag to indicate whether or not to include
In the case of, modulo_time_base and later are included. Note that mo
For data after dulo_time_base, the VOP header and
It is specified that they have the same value, and the information in the VOP header
Information about transmission errors
Has the role of etc. On the other hand, data division refers to FIG.
Indicated by the data structure of the macro block described in
By replacing the code position in the video packet
Important data of each macro block is
This is a function that reduces the probability of being unreadable. FIG. 27 shows a case where the frame type is I-VOP.
About the data division structure in the video packet
Things. Video packet header 24 in FIG.
01 is as shown in FIG. Figure
29 is the data arranged in the priority data section 2402 of FIG.
Data in the priority data section for one macroblock.
1 shows a data structure 24021. In this FIG.
The data is reproduced from the macroblock data shown in FIG.
Extract only important data when synthesizing raw blocks
It can be said that. The priority data section 2402 in FIG.
FIG. 29 shows each macroblock in a video packet.
Only the data shown) are extracted and arranged sequentially. However
However, dquant and Intra difference DC components are shown in the figure.
Exists only in the macroblock for which the condition is satisfied.
Dc_marker 2403 in FIG. 27 is a 19-bit unique word.
Mode. FIG. 30 shows the AC component control information 2404 in FIG.
AC data for one macroblock
This shows the data structure 24041 of the minute control information section.
You. FIG. 31 is arranged in the AC component information 2405 of FIG.
Component information section for one macroblock
3 shows a data structure 24051 of FIG. Either
Also, for each macroblock in the video packet,
29 or FIG. 29 from the macroblock data shown in FIG.
Only the data shown at 30 is extracted and arranged sequentially.
However, the existence of AC component information in each macroblock is cb
It changes depending on the values of py and cbpc. With such a structure
Transmission error at 2404 or 2405
Occurs, the priority data section 2
It can be determined that 402 is decoded without loss of synchronization
Become like As a result, only the data in the priority data
To be used for all macroblocks in the video packet.
And synthesize block images with rough features
Becomes possible. Also, transmission error in priority data section 2402
Even if an error occurs, by searching for dc_marker,
It is also possible to restore synchronization within the video packet.
However, if the data in the priority data section is broken, AC component control
Transmission error in information section 2404 and AC component information section 2405
Mb_type, cbpc, dquant, etc.
Can not always be decoded correctly because the information of
Not exclusively. FIG. 32 shows a case where the frame type is P-VOP.
About the data division structure in the video packet
Things. Video packet header 24 in FIG.
01 is as shown in FIG. Figure
Reference numeral 33 denotes a data located in the priority data section 2902 of FIG.
Data in the priority data section for one macroblock.
1 shows a data structure 29021. In this FIG.
The data is reproduced from the macroblock data shown in FIG.
Extract only important data when synthesizing raw blocks
It can be said that. The priority data section 2902 in FIG.
For each macroblock in a video packet, FIG.
Are extracted and arranged sequentially. However
For a macroblock with a not_coded value of 1, m
cbpc and motion vector information are omitted. Also, mb_type
Becomes an intra-coded or stuffing macroblock
In this case, the motion vector information is omitted. Mo in FIG.
tion_marker 2903 is a 17-bit unique word
is there. FIG. 34 shows the Intra difference DC component information & AC component of FIG.
Regarding the data arranged in the control information 2904, one macro
Intra difference DC component information & AC component control information
It shows the data structure 29041. FIG.
Intra AC component or Inter DC & AC component information 290 in FIG. 32
5, data for one macro block
Intra AC component or Inter DC & AC component information data structure 2
9051 is shown. Both are video packets
For each macro block in the macro, the macro shown in FIG.
The data shown in FIG. 34 or FIG.
Only the data are extracted and arranged sequentially. However, not_coded
For a macroblock of which
Since the data shown in FIGS. 34 and 35 does not exist,
Omitted. In addition, the Intra difference in each macroblock
Ac_pred_f in DC component information & AC component control information 29041
For lag, dquant and Intra difference DC components,
Exists only in macroblocks that satisfy the conditions shown in
I do. In addition, the Intra AC component in each macroblock
(when mb_type is intra coding) or Inter DC & AC component
Information (when mb_type is predictive coding) exists in cbpy, cbpc
Varies depending on FIG. 36 shows the processing in the data correction processing unit 707.
Using video packet and data division functions
Data is shown as an example. First, the error and synchronization shown in FIG.
From the marker detection unit 701, the data search result
F (various start codes or resync_marker) or
Indicates which of the error detection embedded data
Receiving the identification information 713, the detection result is a synchronization marker.
It is determined whether or not it has been (Step 3301). The detection result is
If the marker is a synchronization marker, the position
Data, ie, data for one video packet
From the error / synchronization marker detection unit 701 in FIG.
The second memo is stored in the control information 704 notified to the switch 705.
Output to the memory 710. In the data processing unit 707, the second
Indicates that the input data is stored in the memory 710
The information is notified, and the process ends (step 3313).
On the other hand, the detection result was embedded data for error detection
In this case, the error / synchronization marker detection unit 701 in FIG.
Control information 704 notified to the switch 705
Media data from the search start position to the next synchronization marker
208 is obtained from the memory 207 to the data correction processing unit 707.
(Step 3302). Next, the data processing unit 70
In step 7, the acquired media data 208 is analyzed and error
-Check the section. Check that the error section is VOL
Include header? (Step 3303), Step 33
If the VOL header is not included in 03, the error section is VOP
Include header? (Step 3304), Step 33
04 does not include a VOP header,
Macroblock position data in the video packet header (Figure
Include 28 macroblock_number)? (Step 330
5) If the VOL header was included in step 3303,
If so, is the VOL header a retransmission header? (Step
3306), including a VOP header in step 3304
If so, is vop_coding_type B-VOP? (Step
3307). Here, step 3307
The vop_coding_type information in the VOP header
If the packet is included in the
Vop_coding_type information included in the
_type) may be used for this determination. Error section check
As a result, in step 3306, the error section is
If it is determined that an L header is included,
To discard the header and VOL header combination.
The error correction information 708 is stored in the second memory 710 of FIG.
, And terminates the process (step 3315). this
At this time, the VOS header and VO header are already stored in the second memory 710.
If moving, the second memory contains error correction information
In accordance with 708, the corresponding header information is deleted. Ella
-As a result of the section check, in section 4), the error section
If it is determined to include the first VOL header, an error
Media data up to the next VOL header not included is data
Switch control information to be input to the correction processing unit 707
712 and notifies the switch 705 in FIG.
(Step 3314). And until the next VOL header
Information is deleted and only the VOL header information is corrected.
08 and passed to the second memory. Also, V after the VOL header
If the OP is not an I-VOP, media data until the next I-VOP
Data to the data correction processing unit 707.
Control information 712 to notify the switch 705.
You. And the type of VOP following the VOL header at the beginning of the scene is
Delete data so that it becomes an I-VOP. Error section
As a result of the check, in step 3307, vop_coding_typ
If e can be determined to be a B-VOP, the next VOP header
Media data up to is input to the data correction processing unit 707
As shown in FIG.
Switch 705 and the data up to the next start code
Is deleted (step 3315). Error section check
As a result, in 5), judge that vop_coding_type is not B-VOP
If possible, perform the following processes (1) to (4).
Now (step 3317). (1) Synchronous marker or start command with next header information
Switch control information 71 so that data up to the
2 and notifies the switch 705. (2) Set the current macroblock position (macroblock_number) to 0
And (3) A synchronization marker with header information is included in the frame
In this case, a VOP header is generated from the header information. (4) A synchronization marker with header information is included in the frame.
If not, create a VOP header with vop_coded set to 0.
You. The value of vop_time_increment of the next frame is
Modulo_time_base is set to 10
Then, vop_time_increment is set to 0. Otherwise
Sets modulo_time_base to 0 and vop_time_increment
The value obtained by adding 1 to the value of the previous frame. In addition, processing (3),
When searching for a synchronization marker with header information shown in (4),
The media data 208 in the memory 207 is used. place
When processing (4) is executed, the end of the correction data is
A toe line process is performed (step 3311), and the second
The modified data 708 that has been byte-aligned
The process is output, and the process ends (step 3312). Ella
-As a result of the section check, in step 3305, an error
The section is the macroblock position data in the video packet header.
Data from the second memory,
Read the processed data 709 of the
Investigate the macroblock_number and the number of encoded MBs of the previous VP
(Step 3316). By this processing, the relevant VP head
Macroblock_number can be reproduced. In the error section processing, step 3303 and
When the processing of step 3304 is not satisfied,
At 3307, it is determined that the VOP coding type is not B-VOP.
And the processing (3) is executed in the processing of the step 3317.
The macro block data correction process.
Do. First, the processing of steps 3303 and 3304
If this is not the case, the video packet
Detect the position of the first MB (macroblock_number) (step
3308). Step 3304 is satisfied
In the process (2) of step 3317, the video
Since the position of the first MB of the packet is set to 0,
Step 3308 can be omitted. Next, the menu in the memory 207 is
Media data 208, and finds the location of the 1st MB of the next VP,
That is, the number of processing MBs in the VP is detected (step
3309). As a result of the search, the next VP's resync_marker
If a start code is detected before the
It is determined that the last MB is the last MB of the VOP. And
Information on the number of coded macroblocks in video packets and video
Using the media data 208 in the packet, FIG.
Of the macro block data correction process 3310 shown in FIG.
You. Finally, byte alignment processing 3 is performed at the end of the correction data.
311 and byte-aligned in the second memory 710
The corrected data 708 is output, and the process is terminated (step
Step 3312). Next, the macro block data correction shown in FIG.
The process 3310 will be described. Each macro in the packet
In this specification, the following data correction method is used for a block.
Three methods will be used. Process I: Set the not_coded flag to 1. Original data in MB
Is deleted. Process II: All differential DC components in MB are set to 0, and mb_type is set to
Set “intra”, cpby, cbpc without coding block
You. Original data in MB is deleted. Process III: Set cpby and cbpc without coding block
You. Furthermore, in the case of I-VOP, ac_pred_flag is set to 0.
Then, the AC component data is deleted. For P-VOP, mb_ty
When pe is intra-coded (intra, intra + q), processing I, m
When b_type is predictive coding (inter, inter + q, inter4v)
Deletes the Inter DC & AC component data. The selection procedure of the processing method is shown in FIG.
Determine in minutes if an error has occurred. Therefore,
The media data 208 in the input video packet is
Check the error section and check if the error section is a video packet.
Header (2401 in FIG. 27 or 2401 in FIG. 32)
Including? (Step 3401), Step 3401
Error section if video packet header is not included
Is the priority data section (2402 in FIG. 27 or 2 in FIG. 32).
902)? (Step 3403), Step 3
If the video packet header is included in 403,
-The section is an intermediate marker (2403 in FIG. 27 or FIG. 32).
Contains 2903)? (Step 3404)
Check in three steps. At step 3401, the error section is
If it is determined that the packet includes the
Step 3402 processing is executed, and the processing ends. Specifically
Is first detected in the process of step 3308 in FIG.
Modified the video packet header according to the position of the first MB
I do. At this time, quant_scale is included in the error interval.
In this case, it is set to the same value as the previous VP packet. Next,
Generate data for each MB in the video packet. Generate
As a method, if the coding type of the VOP is I-VOP,
In the case of P-VOP, the method of processing I is used. Sa
In addition, after the priority data section, an intermediate marker (for I-VOP,
dc_marker, motion_marker for P-VOP)
You. Note that in this case, the original data in the packet is
Discard all and output only the correction data. At step 3403, priority is given to the error section.
If it is determined that no data is included, step 34
07, and the process ends. Specifically, DC
T data part (2404 and 2405 in FIG. 27 or FIG. 3
2 2904 and 2905) for the normal MB
Data in the data section and DCT data section are saved as is
I do. The DCT data part is for MB included in the error section.
Then, the data is corrected by the method of processing III. In addition,
The resync_marker of the next VP header is included in the error section
If there is, it is detected in the process of step 3309 in FIG.
The VP that was issued is actually the next VP, and the MB in the next VP is also
It needs to be treated as MB in the VP. Like this
For MBs with no data part information, the VOP encoding
If processing is I-VOP, priority is given to processing II, if P-VOP, processing I
Data section and AC component control information section (only for I-VOP)
Of MB data, and an intermediate marker immediately after the priority data section
Insert At step 3404, the error section is
If it is determined that the marker is included, step 3
The processing of 405 is executed, and the processing ends. In particular,
Processing III is performed for MBs whose priority data section is normal.
Data part of the MB on the I-VOP included in the error section.
Processing II, the priority data part is included in the error section
For MB on P-VOP, data is corrected by method I
I do. Furthermore, immediately after the priority data section, an intermediate marker (I-VOP
Dc_marker in case of, motion_marker in case of P-VOP)
Insert At step 3404, the error section is
If it is determined that no marker is included, step 34
06 is performed, and the process is terminated. Specifically,
If the MB is normal for both the destination data section and the DCT data section,
The data in the destination data section and DCT data section are
Exist. MB in which only the priority data section is included in the error section
For processing II, if the encoding type of the VOP is I-VOP,
For P-VOP, use process I for priority data part and AC component
Correct the MB data in the control information section (only for I-VOP). D
About MB where only CT data part is included in error section
Modifies the data in the manner of process III. The following
Error section up to resync_marker of VP header is included
If there is, it is detected in the process of step 3309 in FIG.
The VP that was issued is actually the next VP, and the MB in the next VP is also
It needs to be treated as MB in the VP. Such priority
For MBs without data part information, VOP encoding type
Is I-VOP, process II, and P-VOP is process I.
M of data part and AC component control information part (only for I-VOP)
Generate B data. As described so far, data error has occurred.
-Correction functions include start code, synchronization marker and intermediate
Performed starting from the marker. Therefore, these units
Keywords, for example, intermediate markers and DCT data
So that the sender part is not included in one packet.
By configuring the transmission packet, it is described in the present invention.
The method works better. The present invention provides the following
Also included. The first is the embedding for error detection of the present invention.
The method of embedding only data is
The packet can be used in the
Packet sequence number (order of packet synthesis)
Connection with the sending terminal
It is applicable regardless of the connection type or connection type.
Therefore, embedding of the error detection embedded data of the present invention
The method is specific to TCP and UDP / RTP as shown in this specification.
Not. Also, even in the case of circuit switching,
Application to correct transmission errors at breaks
The error correction result can be reflected in the output to the decoder.
It is possible. In this way, a specific part of the received data
If it can be determined that an error has occurred, the error
The error detection data embedding method can be applied. The second
Is a method of embedding the error detection embedded data of the present invention.
The method is based on the data error caused by the packet loss part etc.
The position is explicitly communicated to the application decoder.
Special embedding of identification codes into media data.
The specific numerical values are shown in Figs.
It does not depend on the numerical value shown in FIG. Packet loss part
The application decoder that plays back the data
Embedding data generation means and embedding that can be determined
The configuration is included in the present invention. Also, how to detect the embedding position
The method and configuration are not limited to FIGS.
No. Third, the media data correction method of the present invention
For images and sounds affected by transmission errors such as
Input media data to the application decoder
Before decrypting and reproducing the media data.
The feature is that the data is modified to conform to the specifications and standards.
You. Therefore, in this specification, the MPEG-4 video standard is taken as an example.
As described above, such data correction is
The same can be achieved with an encoding method, and various static
Still picture coding method, audio coding method and data coding
Format, as well as media integrated file formats such as MP4.
It is also feasible for mats. Therefore, the application
Video decoder, audio decoder
Decoder, scene description decoder, file format decoder
Data and their combinations).
All methods of converting data to conform to specifications / standards are in accordance with the present invention.
include. The configuration for realizing the present invention is shown in FIG.
It is not limited to eight. For example, with one memory
How to correct and replace data from time to time and error detection
The present invention also includes a case where the function is included in the data correction processing unit.
included. Fourth, in this specification, a mobile phone terminal is used as an example.
As described above, the present invention is applicable to other wireless terminals and distribution
It can also be realized by a server or the like. For example, from wireless communication to wired communication
The data correction method of the present invention is applied to the data converter and converter.
It is effective to apply. According to the present invention, application decoding
Correct the data position affected by packet loss
It is possible to reliably detect. Also, the application
Media data before input to the decoder.
By modifying specifications and standards, application decoding
Data load and reduce the burden of error correction processing
It is possible to increase the speed.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is output data of a system decoder in which a unique word of the present invention is embedded. FIG. 2 is a diagram showing an internal configuration of a mobile phone terminal. FIG. 3 is an example diagram of a transmission packet. FIG. 4 is an example diagram of output media data from a system decoder when no packet loss has occurred. FIG. 5 is an example of output media data from a system decoder when a packet loss occurs. FIG. 6 is a system decoder for implementing embedding processing of embedded data for error detection. FIG. 7 shows output data from the packet processing unit to the media data reconstructing unit in the system decoder of the present invention. FIG. 8 is an example of a processing procedure of a packet processing unit in the system decoder of the present invention. FIG. 9 is a data configuration of an IPv4 packet. FIG. 10 shows a data structure of a PPP frame. FIG. 11 is a data configuration of an RLP frame. FIG. 12 is a data configuration of a TCP packet. FIG. 13 is a data configuration of a UDP packet. FIG. 14 is a data configuration of an RTP packet. FIG. 15 shows an example of error detection embedded data according to the present invention. FIG. 16 shows an example of error detection embedded data to which error notification data is added according to the present invention. FIG. 17 is an internal configuration diagram of a mobile phone terminal having an error notification data output function. FIG. 18 is an internal configuration diagram of a mobile phone terminal that implements data correction processing of the present invention. FIG. 19 is an example of a block diagram of an MPEG-4 decoder. FIG. 20 is an example of a block diagram of an MPEG-4 decoder having an error correction function. FIG. 21 shows an example of an MPEG-4 video encoded bit stream. FIG. 22 is a data configuration of a VOP header in an MPEG-4 video encoded bit stream. FIG. 23 is a diagram showing macroblock division in MPEG-4 video encoding. FIG. 24 shows the configuration of a macroblock in MPEG-4 video encoding. FIG. 25 is a data configuration of MB data in an MPEG-4 video encoded bit stream. FIG. 26 is an example of video packet division in MPEG-4 video encoding. FIG. 27 shows a data structure of video packet data (I-VOP) in an MPEG-4 video encoded bit stream. FIG. 28 is a data structure of a video packet header in an MPEG-4 video encoded bit stream. FIG. 29 is a data structure of a video packet priority data section (I-VOP) in an MPEG-4 video encoded bit stream. FIG. 30 is a data structure of a video packet AC component control information section in an MPEG-4 video encoded bit stream. FIG. 31 is a data structure of a video packet AC component information part in an MPEG-4 video encoded bit stream. FIG. 32 shows a data structure of video packet data (P-VOP) in an MPEG-4 video encoded bit stream. FIG. 33 shows a data structure of a video packet priority data part (P-VOP) in an MPEG-4 video encoded bit stream. FIG. 34 shows a data structure of a video packet Intra difference DC component information & AC component control information section in an MPEG-4 video encoded bit stream. FIG. 35 shows a data structure of a video packet Intra AC component or an Inter DC & AC control information section in an MPEG-4 video encoded bit stream. FIG. 36 is a flowchart showing a media data correction algorithm according to the present invention. FIG. 37 is a flowchart showing a macroblock data correction algorithm according to the present invention. FIG. 38 is a diagram showing the internal configuration of the mobile phone terminal when using RTP. FIG. 39 is a diagram showing an internal configuration of a mobile phone terminal when using TCP. FIG. 40 is a diagram showing an internal configuration of a mobile phone terminal capable of performing a retransmission determination of an illegal packet in consideration of a processing situation in an application decoder. FIG. 41 is a system decoder that embeds error detection embedded data. FIG. 42 is an example of a processing procedure of a packet processing unit in pseudo streaming in the system decoder of the present invention. FIG. 43 is an internal configuration diagram of a mobile phone terminal having an output function of error notification data when using RTP. FIG. 44 is an internal configuration diagram of a mobile phone terminal having a function of outputting error notification data when using TCP. [Description of Signs] 205: System decoder, 207, 710: Memory,
209: application decoder, 301: packet processing unit, 303: retransmission determination result / sequence number, 30
4 retransmission determination unit, 305 response confirmation unit, 307 payload data, 308 media data reconstruction unit, 801
.., Time stamp, sequence number, payload type, 802, packet loss detector, 701, error / synchronous marker detector, 805, 8051, 8052, 805
3: embedded data for error detection, 806, 807: sequence number and payload type in which packet loss has occurred, 807: sequence number in which packet loss has occurred, 706: error notification data, 705: switch, 7
07: data correction processing unit, 3310: macro block data correction unit, 3311: byte alignment processing unit

   ────────────────────────────────────────────────── ─── Continuation of front page    (72) Inventor Junichi Kimura             1-280 Higashi Koigakubo, Kokubunji-shi, Tokyo             Central Research Laboratory, Hitachi, Ltd. F term (reference) 5C059 KK01 MA05 MA23 MC11 NN01                       RB02 RC04 RC22 RF02 RF07                       RF09 RF23 SS06 TA76 TC22                       TD13 UA05 UA33                 5K014 AA01 FA02 FA09 FA15                 5K034 AA02 AA05 CC02 DD01 EE03                       EE11 FF13 HH10 KK28 MM01                       SS03 TT01

Claims (1)

  1. Claims: 1. A packet processing unit for extracting a payload portion from a plurality of input packet data, and a sequence number of packet data that has not arrived or a packet data accompanied by a transmission error is detected. A packet loss detection unit that outputs a unique word for error detection in place of a payload portion of a packet that has not arrived and a packet with a transmission error corresponding to the sequence number that has arrived, and an application decoder that outputs the payload and the unique word. And a media data reconstructing unit that outputs the data in the order of reproduction. 2. The system decoder according to claim 1, wherein the unique word is a data string not included in the specification of the application decoder, and is a data string determined by the application decoder as an error. 3. The system decoder device according to claim 1, wherein said unique word is a data string previously determined as error notification data with an application decoder. 4. An error notification output unit receiving an input from the packet loss detection unit and outputting error notification data having the same size as the media data output from the media data reconfiguration unit. 2. The system decoder device according to claim 1, wherein: 5. The packet loss detector detects the sequence number of the non-arriving packet data and the packet data with a transmission error, and simultaneously detects the payload type of the packet, and determines the payload type. 2. The system decoder according to claim 1, wherein a different unique word is output for each type of media decoder. 6. The packet loss detection unit detects the payload type of the packet when detecting the non-arriving packet data and the sequence number of the packet data accompanied by a transmission error. 2. The system decoder device according to claim 1, wherein the error decoder outputs different error notification data for each type of media decoder determined as the payload type by the payload type. 7. A packet processing unit for extracting a payload portion from a plurality of input packet data, a retransmission determination as to whether to retransmit packet data that has not arrived and packet data with a transmission error, and output the result. A retransmission determination unit, detects the sequence number of the packet data that has not arrived and the packet data accompanied by the transmission error, outputs the sequence number to the retransmission determination unit, A packet loss detection unit that outputs a unique word for error detection in place of a payload portion of a packet accompanied by a transmission error; and media data reconstruction that outputs the payload portion and the unique word in the order in which the application decoder reproduces them. And a system decoder. 8. The retransmission determination unit determines the necessity of retransmission of unarrived packet data and packet data accompanied by a transmission error by using information relating to a processing situation in an application decoder. The system decoder device according to claim 7, wherein 9. The retransmission judging unit, based on the importance of the non-arriving packet data and the packet data with a transmission error, necessity of retransmitting the non-arrival packet data and the packet data with a transmission error. 8. The system decoder according to claim 7, wherein 10. A payload portion is extracted from a plurality of input packet data, a sequence number of packet data that has not arrived and a packet number with a transmission error is detected, and a sequence number of the packet that has not arrived and a packet with a transmission error is detected. A packet that outputs a unique word for error detection instead of the payload portion, generates a signal in which the payload portion and the unique word are replaced in the order in which the application decoder reproduces the packet, and detects and corrects the unique word from the signal. How to correct the data. 11. The method according to claim 10, wherein said correction is performed by correction means including a byte alignment process. 12. When the signal is video data generated by a block-based coding algorithm, a process of changing the coding type information included in the data of the coded block is used to correct the data. The method for modifying packet data according to claim 10, further comprising: 13. The packet data correction method according to claim 10, wherein when the signal is video data, the correction of the data includes a process of changing time stamp information included in the data of the frame image. . 14. When the signal is video data generated by a block-based coding algorithm, the data is corrected by changing a coded block to a block without coded data and a code associated with the block. 11. The method according to claim 10, further comprising the step of changing a flag indicating presence or absence of encrypted data. 15. When the signal is video data generated by a block-based coding algorithm capable of treating a plurality of coded blocks as one video packet, the data correction is performed by modifying the video packet. The method according to claim 10, further comprising a process of changing block position information included in the header portion. 16. If the signal is video data generated by a block-based coding algorithm capable of treating a plurality of coded blocks as one video packet, a video packet 11. The method according to claim 10, further comprising the step of changing the number of coded blocks to be processed in the packet data.
JP2001208680A 2001-07-10 2001-07-10 Data correction apparatus and data correction method Active JP3931595B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2001208680A JP3931595B2 (en) 2001-07-10 2001-07-10 Data correction apparatus and data correction method

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2001208680A JP3931595B2 (en) 2001-07-10 2001-07-10 Data correction apparatus and data correction method
US09/940,576 US7131048B2 (en) 2001-07-10 2001-08-29 Apparatus for system decoder and method for error correction of packet data
KR20010052915A KR100761181B1 (en) 2001-07-10 2001-08-30 System decoder device and packet data correcting method

Publications (3)

Publication Number Publication Date
JP2003023413A5 JP2003023413A5 (en) 2003-01-24
JP2003023413A true JP2003023413A (en) 2003-01-24
JP3931595B2 JP3931595B2 (en) 2007-06-20

Family

ID=19044458

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2001208680A Active JP3931595B2 (en) 2001-07-10 2001-07-10 Data correction apparatus and data correction method

Country Status (3)

Country Link
US (1) US7131048B2 (en)
JP (1) JP3931595B2 (en)
KR (1) KR100761181B1 (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005006340A (en) * 2003-06-13 2005-01-06 Microsoft Corp Time-oriented best-effort fill-up retry method and system for network communication
JP2007295535A (en) * 2006-03-08 2007-11-08 Canon Inc Method and apparatus for receiving an image undergoing loss during transmission
JP2008514164A (en) * 2004-09-22 2008-05-01 クゥアルコム・インコーポレイテッドQualcomm Incorporated Video demultiplexer and video decoder with effective data recovery
JP2008546230A (en) * 2005-05-13 2008-12-18 クゥアルコム・インコーポレイテッドQualcomm Incorporated Apparatus and method for error recovery using in-band error patterns
JP2009290567A (en) * 2008-05-29 2009-12-10 Canon Inc Data transmission device and data reception device
JP2010517395A (en) * 2007-01-26 2010-05-20 トムソン ライセンシングThomson Licensing Packet exchange transmission method of media data and media data processing method
JP2010193327A (en) * 2009-02-20 2010-09-02 Nec Engineering Ltd Image decoding apparatus and packet loss compensation method
JP2010232861A (en) * 2009-03-26 2010-10-14 Sony Corp Information processing apparatus, sound signal processing method, and program
US8379733B2 (en) 2006-09-26 2013-02-19 Qualcomm Incorporated Efficient video packetization methods for packet-switched video telephony applications

Families Citing this family (54)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2003046912A1 (en) * 2001-11-29 2003-06-05 Sharp Kabushiki Kaisha Data recording method, data deletion method, data display method, recording apparatus, recording medium, and program
FR2833796B1 (en) * 2001-12-19 2004-04-09 Thomson Licensing Sa Method and device for compressing video data encoded by video packets
US7106757B2 (en) * 2001-12-19 2006-09-12 Intel Corporation System and method for streaming multimedia over packet networks
KR100460970B1 (en) * 2002-01-10 2004-12-09 삼성전자주식회사 Data transmitting/receiving system and method thereof
US6983408B2 (en) * 2002-03-08 2006-01-03 Microsoft Corporation Managing error/status information generated during video processing
JP4355156B2 (en) 2002-04-16 2009-10-28 パナソニック株式会社 Image decoding method and image decoding apparatus
US7450646B2 (en) * 2002-06-04 2008-11-11 Panasonic Corporation Image data transmitting apparatus and method and image data reproducing apparatus and method
JP2004038575A (en) * 2002-07-03 2004-02-05 Sony Corp Data transmitting and receiving system, data transmitting and receiving method, information providing device, information providing method, data transmitting device, and data receiving method
JP3821086B2 (en) * 2002-11-01 2006-09-13 ソニー株式会社 Streaming system, streaming method, client terminal, data decoding method, and program
US7627042B2 (en) * 2003-01-23 2009-12-01 Ittiam Systems (P) Ltd. System, method, and apparatus for error recovery in coded video signals
JP4344951B2 (en) * 2003-02-17 2009-10-14 日本電気株式会社 Media encoded data transmission method, apparatus, and program
US7206577B2 (en) * 2003-05-06 2007-04-17 Nokia Corporation Method and apparatus for receiving site selection diversity transmit (SSDT) signal in a wideband code division multiple access (WCDMA) system
GB0321641D0 (en) * 2003-09-16 2003-10-15 Agilent Technologies Inc Methods and apparatus for measuring service disruption
KR100608715B1 (en) * 2003-09-27 2006-08-04 엘지전자 주식회사 SYSTEM AND METHOD FOR QoS-QUARANTED MULTIMEDIA STREAMING SERVICE
US20050100098A1 (en) * 2003-10-23 2005-05-12 Gong-Sheng Lin Highly integrated mpeg-4 video decoding unit
US7567584B2 (en) * 2004-01-15 2009-07-28 Panasonic Corporation Multiplex scheme conversion apparatus
TWI270774B (en) * 2004-01-20 2007-01-11 Mediatek Inc Memory control method and related device
JP2005210219A (en) * 2004-01-20 2005-08-04 Sony Corp Transmitting / receiving system, transmission apparatus and method, receiving apparatus and method, recording medium, and program
US9197857B2 (en) * 2004-09-24 2015-11-24 Cisco Technology, Inc. IP-based stream splicing with content-specific splice points
JP2006129018A (en) * 2004-10-28 2006-05-18 Fujitsu Ltd Radio communication device and mobile station
US7747921B2 (en) * 2005-08-05 2010-06-29 Sony Corporation Systems and methods for transmitting data over lossy networks
JP4322851B2 (en) * 2005-08-19 2009-09-02 Necディスプレイソリューションズ株式会社 Video distribution system and video distribution server
JP4725262B2 (en) * 2005-09-14 2011-07-13 富士フイルム株式会社 Image forming apparatus
EP1827009A1 (en) * 2006-02-28 2007-08-29 Matsushita Electric Industrial Co., Ltd. Video encoder and decoder for an improved zapping service for mobile video reception
KR100728038B1 (en) * 2006-03-03 2007-06-07 삼성전자주식회사 Method and apparatus for transmitting data on plc network by aggregating data
JP4419023B2 (en) * 2006-03-23 2010-02-24 株式会社カシオ日立モバイルコミュニケーションズ Mobile communication terminal and program
US8358704B2 (en) * 2006-04-04 2013-01-22 Qualcomm Incorporated Frame level multimedia decoding with frame information table
CN101087438A (en) * 2006-06-06 2007-12-12 安捷伦科技有限公司 System and method for computing packet loss measurement of video quality evaluation without reference
US8306063B2 (en) 2006-08-29 2012-11-06 EXFO Services Assurance, Inc. Real-time transport protocol stream detection system and method
US8310920B2 (en) * 2007-03-02 2012-11-13 Saratoga Data Systems, Inc. Method and system for accelerating transmission of data between network devices
DE102007018484B4 (en) * 2007-03-20 2009-06-25 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatus and method for transmitting a sequence of data packets and decoder and apparatus for decoding a sequence of data packets
US7936695B2 (en) 2007-05-14 2011-05-03 Cisco Technology, Inc. Tunneling reports for real-time internet protocol media streams
US8023419B2 (en) 2007-05-14 2011-09-20 Cisco Technology, Inc. Remote monitoring of real-time internet protocol media streams
US7835406B2 (en) 2007-06-18 2010-11-16 Cisco Technology, Inc. Surrogate stream for monitoring realtime media
JP5224731B2 (en) * 2007-06-18 2013-07-03 キヤノン株式会社 Video receiving apparatus and video receiving apparatus control method
US7817546B2 (en) * 2007-07-06 2010-10-19 Cisco Technology, Inc. Quasi RTP metrics for non-RTP media flows
CN101394336B (en) * 2007-09-17 2012-02-22 华为技术有限公司 A packet network transmission frame trau method, system and apparatus
US8966551B2 (en) * 2007-11-01 2015-02-24 Cisco Technology, Inc. Locating points of interest using references to media frames within a packet flow
US9128868B2 (en) * 2008-01-31 2015-09-08 International Business Machines Corporation System for error decoding with retries and associated methods
US8902996B2 (en) 2008-02-26 2014-12-02 Richwave Technology Corp. Adaptive wireless video transmission systems and methods
US9357233B2 (en) * 2008-02-26 2016-05-31 Qualcomm Incorporated Video decoder error handling
US8265171B2 (en) * 2008-02-26 2012-09-11 Richwave Technology Corp. Error resilient video transmission using instantaneous receiver feedback and channel quality adaptive packet retransmission
EP2234314A1 (en) * 2009-03-25 2010-09-29 Thomson Licensing SA A method of managing a packet administration map
US8301982B2 (en) * 2009-11-18 2012-10-30 Cisco Technology, Inc. RTP-based loss recovery and quality monitoring for non-IP and raw-IP MPEG transport flows
CN101729840A (en) * 2009-12-17 2010-06-09 于培宁 Storage processing method utilizing video image characteristic sequence
US8819714B2 (en) 2010-05-19 2014-08-26 Cisco Technology, Inc. Ratings and quality measurements for digital broadcast viewers
CN108366300A (en) * 2011-11-25 2018-08-03 麦克赛尔株式会社 Image received device
WO2013115121A1 (en) * 2012-01-31 2013-08-08 シャープ株式会社 Generation device, reproduction device, data structure, generation method, reproduction method, control program, and recording medium
KR101971623B1 (en) * 2012-05-10 2019-04-23 삼성전자주식회사 Method for contents and user's interactions among multiple devices
TWI540886B (en) * 2012-05-23 2016-07-01 晨星半導體股份有限公司 Audio decoding method and audio decoding apparatus
US9104611B1 (en) * 2012-11-13 2015-08-11 Sprint Communications Company L.P. Management of syntax errors in signaling messages
WO2014194204A1 (en) * 2013-05-31 2014-12-04 Cellular South, Inc. Dba C Spire Wireless Bringing real-time testing field environment into rf labor
KR101610715B1 (en) * 2014-06-11 2016-04-08 한국전자통신연구원 One-way data transmission and reception system, and one-way data transmission and reception method
JP5940231B1 (en) * 2016-01-15 2016-06-29 株式会社 ディー・エヌ・エー Information processing system, information processing program, and information processing method

Family Cites Families (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR940012945A (en) * 1992-11-26 1994-06-24 윤종용 Error signal generating method and apparatus
US5737022A (en) * 1993-02-26 1998-04-07 Kabushiki Kaisha Toshiba Motion picture error concealment using simplified motion compensation
JP2576776B2 (en) 1993-11-10 1997-01-29 日本電気株式会社 Packet transmission method, packet transmission device
US5475688A (en) 1994-04-22 1995-12-12 Thomson Consumer Electronics, Inc. Media error code generation as for a video inverse transport processor
JP2898212B2 (en) 1994-12-21 1999-05-31 沖電気工業株式会社 Image decoding circuit
JPH08191337A (en) * 1995-01-09 1996-07-23 Daikin Ind Ltd Device and method for data transmission
US5898695A (en) 1995-03-29 1999-04-27 Hitachi, Ltd. Decoder for compressed and multiplexed video and audio data
JPH09247132A (en) * 1996-03-08 1997-09-19 Yazaki Corp Radio packet communication equipment and transmitting device
US5802050A (en) * 1996-06-10 1998-09-01 Telefonaktiebolaget Lm Ericsson Minicell sequence number count
JP3291532B2 (en) 1996-07-24 2002-06-10 松下電器産業株式会社 Decoder
JP3884172B2 (en) * 1997-10-02 2007-02-21 株式会社東芝 Variable length decoding apparatus and decoding method
US5956102A (en) * 1997-11-04 1999-09-21 Hitachi America Ltd. Methods and apparatus for the efficient implementation of signal synchronization and cyclic redundancy checks in communication systems
JPH11331261A (en) 1998-05-20 1999-11-30 Kokusai Electric Co Ltd Packet communication equipment
US6310884B1 (en) * 1998-05-21 2001-10-30 Lsi Logic Corporation Data transfer method and apparatus that allocate storage based upon a received relative offset
FI105962B (en) 1998-06-30 2000-10-31 Nokia Mobile Phones Ltd Error detection multiplexed signals are being
GB2345829B (en) * 1998-11-02 2001-01-10 Samsung Electronics Co Ltd Video coding and decoding methods
EP1919117B1 (en) 1998-11-30 2014-10-15 Panasonic Corporation Packet retransmission control using priority information
JP4218112B2 (en) 1999-02-26 2009-02-04 三菱電機株式会社 Multimedia communication system
JP3411234B2 (en) 1999-04-26 2003-05-26 沖電気工業株式会社 Encoded information receiving and decoding device
JP2001025010A (en) 1999-07-09 2001-01-26 Mitsubishi Electric Corp Multi-media information communication equipment and method therefor
KR20010063821A (en) * 1999-12-24 2001-07-09 서평원 Detection Apparatus and Method for transmit fail in mobile communication system

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005006340A (en) * 2003-06-13 2005-01-06 Microsoft Corp Time-oriented best-effort fill-up retry method and system for network communication
JP4694152B2 (en) * 2003-06-13 2011-06-08 マイクロソフト コーポレーション Time-oriented, best-effort, hole-filling retry method and system for network communication
JP2008514164A (en) * 2004-09-22 2008-05-01 クゥアルコム・インコーポレイテッドQualcomm Incorporated Video demultiplexer and video decoder with effective data recovery
JP2011172245A (en) * 2004-09-22 2011-09-01 Qualcomm Inc Video demultiplexer and decoder using efficient data recovery
JP4819815B2 (en) * 2004-09-22 2011-11-24 クゥアルコム・インコーポレイテッドQualcomm Incorporated Video demultiplexer and video decoder with effective data recovery
JP2008546230A (en) * 2005-05-13 2008-12-18 クゥアルコム・インコーポレイテッドQualcomm Incorporated Apparatus and method for error recovery using in-band error patterns
JP2007295535A (en) * 2006-03-08 2007-11-08 Canon Inc Method and apparatus for receiving an image undergoing loss during transmission
JP4614395B2 (en) * 2006-03-08 2011-01-19 キヤノン株式会社 Method and apparatus for receiving a lost image during transmission
US8379733B2 (en) 2006-09-26 2013-02-19 Qualcomm Incorporated Efficient video packetization methods for packet-switched video telephony applications
JP2010517395A (en) * 2007-01-26 2010-05-20 トムソン ライセンシングThomson Licensing Packet exchange transmission method of media data and media data processing method
JP2009290567A (en) * 2008-05-29 2009-12-10 Canon Inc Data transmission device and data reception device
JP2010193327A (en) * 2009-02-20 2010-09-02 Nec Engineering Ltd Image decoding apparatus and packet loss compensation method
JP2010232861A (en) * 2009-03-26 2010-10-14 Sony Corp Information processing apparatus, sound signal processing method, and program
US8676363B2 (en) 2009-03-26 2014-03-18 Sony Corporation Information processing apparatus, audio signal processing method, and program product

Also Published As

Publication number Publication date
KR100761181B1 (en) 2007-09-21
KR20030006881A (en) 2003-01-23
US7131048B2 (en) 2006-10-31
JP3931595B2 (en) 2007-06-20
US20030014705A1 (en) 2003-01-16

Similar Documents

Publication Publication Date Title
US7003033B2 (en) Systems and methods for encoding redundant motion vectors in compressed video bitstreams
RU2419245C2 (en) Systems and methods for channels switching
TWI280059B (en) Picture coding method
US8730953B2 (en) Method and system for communicating video data in a packet-switched network, related network and computer program product therefor
US6317462B1 (en) Method and apparatus for transmitting MPEG video over the internet
US8705573B2 (en) Apparatus and method for coding an information signal into a data stream, converting the data stream and decoding the data stream
JP4949591B2 (en) Video error recovery method
DE60110303T2 (en) Method and apparatus for packet transmission with packet header compression
TWI423679B (en) Scalable video coding and decoding
EP1595404B1 (en) Picture decoding method
EP2148513B1 (en) Video coding
JP5007322B2 (en) Video encoding method
Wenger et al. Error resilience support in H. 263+
US7327791B1 (en) Video decoding method performing selective error concealment and resynchronization
RU2385541C2 (en) Variation of buffer size in coder and decoder
Wang et al. Review of error resilient coding techniques for real-time video communications
EP0844792A2 (en) Method for arranging compressed video data for transmission over a noisy communication channel
CN101175213B (en) Video source coding method and device, method and device for decoding video source
JP4612171B2 (en) Video decoding / playback module, playback time management program, and multimedia information receiving apparatus
KR100612003B1 (en) Apparatus for transmitting/receiving bitstream in network and method thereof
EP1157562B1 (en) Video coding
KR100984368B1 (en) Video coding
CN100472997C (en) Error detecting method of mobile communication system
FI115589B (en) Encoding and decoding redundant images
CN1253014C (en) Image coding method and apparatus an image decoding method and apparatus

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20040825

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20040825

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20060213

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20060314

RD01 Notification of change of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7421

Effective date: 20060418

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20060515

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20060725

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20060921

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20070220

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20070305

FPAY Renewal fee payment (prs date is renewal date of database)

Free format text: PAYMENT UNTIL: 20110323

Year of fee payment: 4

FPAY Renewal fee payment (prs date is renewal date of database)

Free format text: PAYMENT UNTIL: 20110323

Year of fee payment: 4

FPAY Renewal fee payment (prs date is renewal date of database)

Free format text: PAYMENT UNTIL: 20120323

Year of fee payment: 5

FPAY Renewal fee payment (prs date is renewal date of database)

Free format text: PAYMENT UNTIL: 20130323

Year of fee payment: 6