JP3323057B2 - Encoding device, decoding device, and transmission system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

TECHNICAL FIELD The present invention relates to heat transmission system, a mark Goka device及beauty decrypted device, for example, it is capable of applying to the improvement to an image quality degradation due to frame loss or lapse in the image transmission .

[0002]

2. Description of the Related Art In recent years, a video telephone system, a video conference system, or a video on demand (VOD)
2. Description of the Related Art Transmission systems for image signals using a network as a transmission path, such as a Demand (Digital Demand) system, are becoming widespread, and with this, international standardization of image encoding methods is being promoted.

[0003] Image coding methods can be classified into two types, those using both intra-frame coding and inter-frame coding and those using only intra-frame coding.

[0004] Among them, the one using both the intra-frame coding method and the inter-frame coding method is a coding method for moving picture communication / storage such as ITU-T Recommendation H.261 or MPEG (Moving Picture Experts Group). The method has been applied. FIG. 2 shows an example of frames arranged in time order. In ITU-T Recommendation H.261, as shown in FIG. 2, periodically intra-frame coding (I frames a, i) is carried out, and during the frame time coded frames (P frames b to h,
By performing inter-frame coding using the previous frame for j-) as a reference image, redundancy related to temporal change is removed. Hereinafter, the intra-coded frame is referred to as I
A frame is referred to as a frame, and an inter-coded frame is referred to as a P frame.

On the other hand, the one using only intra-frame coding is based on JPEG (Joint Photographic Coding Experts).
Group) and other continuous image coding methods, and
, Intra-frame encoding is performed on all frames.

[0006]

In H.261, since inter-frame coding is performed with reference to a previous frame, all frames need to be transmitted reliably and in order. Telephone lines and IS
When data transmission is performed after establishing a line with the other party such as a DN line, there is no problem because the data is sequentially delivered to the other party without being lost on the way.
In AN or ATM, since a line is not established and the data is divided and transmitted in small data units (packets or cells), packets may be lost in the middle or the order may be changed due to a difference in route.

In general, in order to solve this, the transmitting side adds a serial number to the packet and transmits the packet. Protocol (TC)
P: Transmission Control Protocol, etc.) to improve network reliability.

However, in the case where the operation of the network is unstable and packets are frequently dropped, retransmission processing using the above-mentioned protocol accumulates delay, which is unsuitable for real-time transmission of moving images. It is. In particular, if new image data can be displayed, it may be better to display new data even when frames are dropped, rather than retransmitting and displaying old data.

[0009] Broadcast or multicast used when transmitting data to multiple points is a mechanism for transmitting data to a plurality of points in one packet transmission. However, in this mechanism, if a packet is lost at the time of transmitting a packet to one point, and a retransmission process such as the above-described protocol is performed, the network at the other point may not have received the first packet normally. Nevertheless, the same packet must be retransmitted, which significantly increases the load on the network. Therefore, in broadcast and multicast, it is common to use a protocol that does not perform retransmission processing (such as UDP: User Datagram Protocol), and the probability of packet loss is high.

[0010] When a wireless network is used, not only when a packet is divided and transmitted, but also when a line is established and transmitted, the data error rate or loss rate tends to increase. . In particular, even if an error is detected on the receiving side, if an error that is higher than the error correction capability is present, a method of discarding data in a certain section in order to process other parts normally is adopted. The gap is large compared to wired networks.

Another problem is that the processing speeds of the transmitting and receiving terminals do not always match. For example, when the processing speed of the receiving terminal is slower, when all the image frames are decoded and displayed, the frame data waiting to be processed is accumulated, and the delay increases. Therefore, in such a case, it is necessary for the receiving side to adaptively drop frames. However, when a frame is inter-coded by a conventional method, there is a problem that if there is no decoded data of the previous frame, the current frame cannot be decoded, and frame dropping cannot be performed freely.

FIG. 4 shows an example of frame loss in the case of H.261. If the frame e is lost in the middle or cannot be decoded because the processing cannot be completed in time, the next I frame i
Until A arrives, the P frame (f, g, h) in between
Cannot be decrypted.

Therefore, conventionally, in order to reliably transmit all frames in a network in which such frame loss or frame loss frequently occurs, all frames are simply encoded by intra-frame coding without using inter-frame coding. JP to be transmitted
A method like EG was adopted. For example, in the case of the JPEG encoding method shown in FIG. 5, even if the frame e is lost, decoding can be normally performed from the next frame.
However, in this case, since the inter-frame coding is not performed, there is a problem that the redundancy of the temporal change is not removed, the coding efficiency is low, and the transmission data amount is large.

As described above, even in an environment in which frame loss or frame dropout is expected, a P frame can be decoded without waiting for an I frame to be sent later. There is a need for a transmission system, an encoding device, and a decoding device that can suppress a decrease in encoding efficiency even during transmission.

[0015]

In order to solve the above-mentioned problems, according to the present invention, data to be transmitted at a certain point in time is used as reference data used as a reference in predictive coding, and data to be transmitted after each point in time is transmitted. Predictively encode transmitted data,
The encoding apparatus for transmitting the encoded data to the decoding apparatus is characterized in that the following is performed.

That is, an estimating means for estimating the receiving ability of the decoding device based on the reception result of the encoded data at each time point notified from the decoding device, and Reference data updating means for adaptively updating reference data used as a reference at the time of predictive encoding is provided.

In the encoding apparatus according to the present invention, the reference data is updated according to the receiving capability of each decoding apparatus. Therefore, an appropriate reference data is always selected as compared with the transmitted data to be transmitted. And encoding according to the transmission target can be realized. Further, the chances of selecting inappropriate data as reference data can be reduced, so that coding efficiency can be kept high.

According to the present invention, a decoding device for sequentially decoding encoded data received via a transmission path is characterized as follows.

That is, each time point of the encoded data
Status notification to notify the encoding device of the
A means of informing is provided.

In the decoding device according to the present invention, the encoding device side is notified of the encoded data at each point in time.
Since the reception state can be notified, adaptive encoding processing in the encoding device can be realized.

Then, a transmission system is constructed by using the encoding device and the decoding device, and a system with high encoding efficiency is realized.

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a first embodiment in which the present invention is applied to multipoint transmission of a moving image will be described with reference to the drawings.

(A) First Embodiment The transmitting terminal of the transmission system according to the present embodiment informs the transmitting terminal that a frame has been normally received (or abnormal) during multipoint transmission. Decoding error presence / absence signal transmitted by each receiving terminal (hereinafter referred to as confirmation signal)
Is used to estimate the receiving capability of each receiving terminal. If there is a difference in receiving capability, the target terminal that will receive the next frame is limited, and the reference frame is selected and updated according to this. It is characterized by the provision of functions.

The receiving terminal of the transmission system according to the present embodiment has a function of not performing decoding processing on encoded data not intended for the own terminal, and a function of changing the receiving capability of the own terminal from the user. And a function of transmitting the request to the transmitting terminal when the request is received.

That is, the transmission system according to the present embodiment is characterized in that it has a function of adaptively updating the reference frame according to the capability of the receiving terminal.

Next, the configuration of the transmitting terminal and the receiving terminal that constitute the transmission system will be described.

(A-1) Configuration of Video Encoding Apparatus 100 FIG. 1 is a functional block diagram showing a configuration example of a video encoding apparatus 100 as an example of a transmission terminal constituting a transmission system according to the present embodiment. It is.

The moving image input unit 101 is means for transferring moving image data input from a camera or the like to the encoding unit 102 for each frame.

The encoding section 102 is means for encoding the input frame data, and sends the data to the decoding section 103 and the encoded data transmitting section 107. When encoding, Intra / I
In accordance with an instruction from the nter determination unit 106, switching between intra-frame coding and inter-frame coding is performed. Where Intra
Represents intra-frame encoding, and Inter represents inter-frame encoding. At the time of inter-frame encoding, encoding is performed with reference to frame data in the reference frame memory 105.

The decoding unit 103 decodes the encoded data again, and stores the decoded frame data in the frame memory 10.
4 is a means for writing together with the frame number.

The frame memory 104 is a medium for storing data for each frame. Every time the reference frame is updated by the reference frame updating unit 113, unnecessary data is deleted.

The reference frame memory 105 is a medium for storing reference frame data used for inter-frame coding, and is updated by the reference frame updating unit 113.

The Intra / Inter judging section 106 is means for judging whether to perform intra-frame coding or inter-frame coding. Normally, intra-frame coding is performed periodically (for example, every 30 frames), and inter-frame coding is performed in other periods. However, a refresh signal is received from a receiving side (a moving image decoding device 200 described later). In this case, the intra-frame coding is forcibly performed. If the encoding is determined to be intra-frame encoding, the encoding unit 102 and the reference frame updating unit 1
13 to that effect.

The code data transmitting section 107 is means for transmitting the encoded moving image data to the receiving side (the moving image decoding device 200). The data is multiplexed with a flag indicating Intra / Inter, a reference frame number, and the like. Also, transmission source information and transmission destination information are multiplexed and transmitted as necessary. When multipoint transmission is performed, all data is transmitted to all receiving terminals. When limited to some terminals from the target terminal limiting unit 111, transmission may be performed only to that terminal. Alternatively, the target terminal information may be multiplexed and transmitted.

The acknowledgment signal receiving section 108 is means for receiving an acknowledgment signal indicating which frame the receiving side (moving picture decoding apparatus 200) could decode, and passes the confirmed frame number to the reference frame updating section 113. If multipoint transmission is being performed, the multipoint reception recording unit 109
Pass to

The multipoint receiving and recording unit 109 records the past reception results such as which frame each receiving terminal has received up to now and the propagation delay from transmitting the encoded data to receiving the confirmation signal. This is a means for recording for each terminal, and passes the recorded data to the reception capability estimating unit 110, the target terminal limiting unit 111, and the reference frame selecting unit 112. When the reference frame selection unit 112 selects a frame received by all receiving terminals as a reference frame, old data before that may be deleted, or a certain period may be determined and stored. good.

The reception capability estimating unit 110 calculates the average frame loss probability of each receiving terminal based on the past reception results recorded in the multipoint reception recording unit 109, and receives the reception capability (frame rate, network (E.g., propagation delay), and passes the estimation result to the target terminal limiting unit 111. The capability estimation may be performed only once after a predetermined time from the start of communication, or may be performed at regular intervals when there is a change in the reception capability or the state of the network. Further, when a receiving capability change request is received from the receiving side (video decoding device 200), the receiving capability is estimated in accordance with the instruction. However, when it is presumed that the reception capability change request from the reception side is difficult to be realized, the fact may be notified to the reception side and the capability change may not be performed.

The target terminal limiting unit 111 limits the terminal that receives the next frame based on the reception performance recorded in the multipoint reception recording unit 109 based on the reception capability of each terminal estimated by the reception capability estimation unit 110. It is a means to do. Normally, all terminals are to be received, but if there is a clear difference in receiving capability and there is a terminal that is not expected to receive the next frame, the target terminal is limited, and the limited target terminal data is used as the reference frame. At the same time as passing to the selecting unit 112, it passes to the code data transmitting unit 107.

The reference frame selecting section 112 is a means for selecting the most commonly received frame as a reference frame based on the reception results of the terminals limited by the target terminal limiting section 111. To the reference frame updating unit 113.

The reference frame updating unit 113 performs the processing of Intra / Int
This is a means for updating the reference frame according to an instruction from the er determining unit 106, the confirmation signal receiving unit 108, and the reference frame selecting unit 112. When an instruction for intra-frame encoding is received from the Intra / Inter determination unit 106, the data of the current frame is copied from the frame memory 104 to the reference frame memory 105, and all the frame data in the frame memory 104 is deleted. When receiving the frame number from the acknowledgment signal receiving unit 108, the frame data of that number is copied from the frame memory 104 to the reference frame memory 105, and the frame data before that number is copied to the frame memory 1
Erase from 04. When multipoint transmission is being performed, the frame data of that number is copied from the frame memory 104 to the reference frame memory 105 in accordance with an instruction from the reference frame selection unit 112. At this time, the frame data before that number is deleted from the frame memory 104 only when the target terminal is not limited.

(A-2) Configuration of Video Decoding Device 200 FIG. 6 is a functional block diagram showing a configuration example of the video decoding device 200 as an example of a receiving terminal configuring the transmission system according to the present embodiment. It is.

The coded data limited receiving unit 201 transmits the coded moving image data to the transmitting side (the moving image coding apparatus 100).
Means for receiving from The coded data and a flag indicating Intra / Inter multiplexed on the data, a reference frame number, and the like are passed to the decoding unit 206, and the multiplexed transmission source information and the like are also passed to the decoding unit 206 as necessary. Also,
The reference frame number is passed to the reference frame comparison unit 202.
Further, when the target terminal information is multiplexed at the time of multipoint transmission, the information is also transferred to the reference frame comparing unit 20.
Hand over to 2. Here, if the own terminal is not targeted, the received encoded data is discarded, and the decoding process is not performed on the frame.

The reference frame comparing unit 202 compares the received reference frame number with the reference frame number of the own terminal stored in the reference frame memory 204. If the reference frame number is different, the reference frame updating unit 203
To the reference frame update request and pass the new reference frame number. If there is target terminal information, it is also passed.

Upon receiving an update request from the reference frame comparing unit 202, the reference frame updating unit 203 reads data of a new reference frame number from the frame memory 205, and writes and updates the data with the frame number in the reference frame memory 204. At this time, if the target terminal is not limited, old frame data in the frame memory is deleted.

The reference frame memory 204 is a medium for storing reference frame data used for inter-frame decoding, and is updated by the reference frame updating unit 203 and the decoding unit 206.

The frame memory 205 is a medium for storing the decoded frame data.

The decoding unit 206 is a means for decoding the input coded data, and sends it to the decoded data moving image output unit 208. At the time of decoding, in the case of an I frame, the decoded data is stored in the reference frame memory 2 together with the frame number.
04 to update the reference frame. In the case of a P frame, decoding is performed with reference to the data in the reference frame memory 205, and the decoded data is written into the frame memory 205 together with the frame number. If the decoding is successful, the frame number is passed to the confirmation signal transmitting unit 207.

Confirmation signal transmitting section 207 transmits a confirmation signal indicating which frame could be decoded to the transmitting side (moving picture coding apparatus 1).
00), and also transmits the frame number received from the decoding unit 206.

The moving image output section 208 is means for outputting the decoded data received from the decoding section 206 to a monitor or the like.

The refresh signal transmitting section 209 is means for transmitting a refresh signal to the transmitting side (moving picture coding apparatus 100) in accordance with a refresh request from a user.

The receiving capability change requesting means 210 receives a request from the user to change the receiving capability of the receiving terminal estimated on the transmitting side.
0) is a means for sending a reception capability change request. Here, if the request from the user exceeds the capability of the receiving terminal, it may be notified to the user and the change request may not be sent.

(A-3) Operation at Multipoint Transmission Next, an example of the operation of the video encoding device 100 and the video decoding device 200 at the time of multipoint transmission will be described. Here, the operation states of the two devices 100 and 200 will be described separately in two periods, a period after communication starts and a period after that. In this description,
It is assumed that the number of moving picture decoding apparatuses 200 connected to the moving picture coding apparatus 100 is two, and they are represented as receiving terminals A and B, respectively.

First, an operation example for a while after the start of communication will be described. At this time, the moving picture coding apparatus 100 has not yet grasped the capability of the receiving terminal connected to the moving picture coding apparatus. At this time, based on the confirmation signal (or the refresh signal) transmitted for each of the receiving terminals A and B, the video encoding device 100 normally receives images for both the receiving terminals A and B. It operates so as to update the reference frame only when the notification is made.

That is, as shown in FIG. 7, the reference frame is updated only for the frames a, e and i, and for each of the other frames b, c, d, f, g, h, j and k, Encoding is performed based on a frame stored as a reference frame at each time point. In the example of FIG. 7, frames b, c, and d are based on frame a, and frames f, g, and h are based on frame e.

Therefore, during this period, the reference frame is updated in accordance with the receiving terminal B having relatively low receiving capability, and each frame is encoded.

As a result, it is possible to eliminate the need for intra-frame encoding of all frames in preparation for a frame loss or the like and to decode a P frame without waiting for a subsequent I frame. Higher transmission efficiency can be obtained as compared with the above. However, if the timings at which both the receiving terminals A and B can receive normally are significantly shifted, or if the difference between the receiving capabilities of the two receiving terminals is large, the number of times of updating the reference frame is expected to decrease. In some cases, encoding using this technique cannot improve encoding efficiency as much as expected.

Therefore, what is used is the past reception results accumulated during a period of time from the start of the communication, and based on the estimation of the reception capability of each of the receiving terminals A and B, the code of the frame is determined. Modify the conversion method.

For estimating the receiving capability, a receiving capability estimating unit 110 is used. In the case of FIG. 7, the reception capability estimating unit 110 indicates that the receiving terminal A can receive almost every frame, but the receiving terminal B has only the ability to receive once every four frames on average. presume.

Based on the result, the target terminal limiting unit 111
Determines that the receiving terminal B cannot receive, for example, the frame i four frames ahead after receiving the frame e, and encodes the frames f, g, and h between them only to the terminal A. Limit the target terminals.

That is, since the receiving terminal A has received every frame, as shown in FIG. 8, the reference frame is also updated every frame, and when the terminal i encodes the frame i which both terminals can determine to receive. Only, the terminal operates so as to return the reference frame to the frame e received in common.

As a result, although the coding efficiency for the receiving terminal A is lower than that in the one-to-one communication, it does not become as bad as the capability of the receiving terminal B. Efficiency is improved.

FIG. 9 shows a case where a frame is lost during the encoding. When a loss occurs (frames c, e, and i in the figure), the acknowledgment signal receiving unit 108 that has received the acknowledgment signal detects this and causes the next frame to be encoded without updating the reference frame. As a result, the image quality is not significantly deteriorated even for the missing frame.

FIG. 10 shows a case where the receiving timings of the two receiving terminals do not match. In this case, since the partner terminal is limited for each frame, the update of the reference frame suitable for each receiving terminal is realized, and only the frame that can be received by both terminals is used as the reference frame (FIG. 11). ), The coding efficiency can be improved.

(A-4) Effects of the First Embodiment According to the above configuration, the updating of the reference frame is controlled and the next frame is encoded according to the reception result of each receiving terminal. All frames can be decoded without waiting for the transmission of the I frame, and the coding efficiency can be improved.

Also, by providing the transmitting terminal with a function for estimating the receiving capability of each receiving terminal, even if the number of receiving terminals increases, frame coding and data transmission suitable for each receiving terminal can be realized. it can.

This is effective at the time of multipoint transmission. Even if receiving terminals having different receiving capacities are connected to one line, the influence of the receiving terminals having low capacities is small.
For a receiving terminal with high capability, high coding efficiency can be maintained.

In multipoint transmission, even if the reception timings of a plurality of receiving terminals are shifted, a reference frame suitable for each terminal can be updated, and high coding efficiency can be maintained. .

(B) Second Embodiment Next, a second embodiment will be described. The difference between the second embodiment and the first embodiment is that, on the transmitting terminal side, the receiving terminals having the same receiving capability are grouped with respect to the plurality of receiving terminals receiving the image transmission from the transmitting terminal. And a means for limiting transmission of frames at the same timing to receiving terminals classified into the same group, which is characterized by reducing the load of transmission processing.

(B-1) Configuration of Video Encoding Apparatus 300 FIG. 12 shows a video encoding apparatus 300 according to the second embodiment.
FIG. 2 is a functional block diagram illustrating a configuration example of the present invention. 12, the same or corresponding parts as those in FIG. 1 are denoted by the same or corresponding reference numerals, and description thereof will be omitted.

Reception ability estimating section 301 has a function equivalent to that of receiving ability estimating section 110 described in the first embodiment. hand over.

Receiving terminal classifying section 302 is a means for grouping the receiving terminals by capability based on the receiving capacity estimated by receiving capacity estimating section 301, and passes the classification result to target terminal limiting section 303.

The target terminal limiting unit 303 has the same function as the target terminal limiting unit 111 described in the first embodiment, and is further classified into the same group based on the classification result of the receiving terminal classifying unit 302. always limited to the same timing relative to the receiving terminals. Thereby, the transmission of the encoded data is always guaranteed to the receiving terminals classified into the same group at the same time.

The structure of the receiving terminal side, that is, the structure of the moving picture decoding apparatus is the same as that of the first embodiment.

(B-2) Operation at the Time of Multipoint Transmission A decrease in the coding efficiency due to the mismatch of the reception timings of the two receiving terminals can also be avoided by the moving picture coding apparatus 100 according to the first embodiment. However, in the case of the moving picture coding apparatus 100, coding and transmission are alternately repeated at the same timing, so that waste is included.

Therefore, in the moving picture coding apparatus 300 according to the second embodiment, as shown in FIG. 13, the receiving terminals A and B having the same capability are grouped and transmitted at the same timing. .

In this way, unnecessary processing on the transmitting terminal side can be omitted, and processing efficiency can be improved.

FIG. 14 shows a case where a frame is lost in this example. Also in this case, when a loss occurs, the encoding of the frame is performed without updating the reference frame. Also,
In the present embodiment, receiving terminals classified into the same group are treated exactly the same.

(B-3) Effects of the Second Embodiment As described above, also in the present embodiment, the same effects as in the first embodiment can be obtained. The following two effects are obtained as effects unique to the form. One is that receiving terminals with similar receiving capabilities are treated the same,
There is no risk that the timing of the encoding and transmission processing will be shifted, and there is no waste of processing. Another is that terminals with the same receiving capacity are treated in the same way, so that the Greatly reduced the number of combinations
That is, simplification of processing can be realized.

(C) Third Embodiment A third embodiment will be described. The difference between the third embodiment and the first embodiment is that, on the transmitting terminal side, for the frame data stored by the transmitting terminal, which receiving terminal was selected as a reference frame in the past in the past , Means for selecting an old frame intended for the combination of the receiving terminal as an unnecessary frame, means for erasing selected frame data from a frame memory, and means for transmitting an unnecessary frame number to a receiving side. And a means for receiving unnecessary frame numbers and a means for deleting unnecessary frames on the receiving terminal side. In view of the above, the present embodiment is characterized in that unnecessary frame data among reference frame candidate frame data can be adaptively deleted according to the capability of the receiving terminal.

(C-1) Configuration of Video Encoding Apparatus 400 FIG. 15 shows a video encoding apparatus 400 according to the third embodiment.
FIG. 2 is a functional block diagram illustrating a configuration example of the present invention. Note that, in FIG. 15, the same or corresponding portions as those in FIG. 1 are denoted by the same or corresponding reference numerals, and description thereof will be omitted.

Reception ability estimation section 401 has the same function as reception ability estimation section 110 described in the first embodiment.
The estimated receiving capability is transferred to the target terminal limiting unit 402 and also to the unnecessary frame selecting unit 404.

The target terminal limiting unit 402 has the same function as the target terminal limiting unit 111 described in the first embodiment,
The limited target terminal data is transferred to the reference frame selecting unit 112 and also to the target terminal recording unit 403.

The target terminal recording unit 403 is a means for recording which terminal the frame data recorded in the frame memory 104 has been selected as a reference frame in the past, and stores the recorded data in the unnecessary frame selection unit 4.
Hand over to 04. Note that the recording of the target terminal, may be a dedicated tape blanking <br/> Le, may be added to the header information of the frame memory 104.

The unnecessary frame selection unit 404 is means for selecting an unnecessary frame from the recorded frame data, which will not be selected as a reference frame in the future, based on the reception capability. To the unnecessary frame erasing unit 405. Also, if the receiving side is also compatible, it passes it to the unnecessary frame number transmitting unit 406.

The unnecessary frame erasing section 405 sends the frame memory 104 according to the instruction of the unnecessary frame selecting section 404.
This is a means for deleting the unnecessary frame data.

The unnecessary frame number transmitting section 406 is means for transmitting an unnecessary frame number to the receiving side as necessary.

(C-2) Configuration of Moving Picture Decoding Apparatus 500 FIG. 16 shows a moving picture decoding apparatus 500 according to the third embodiment.
FIG. 2 is a functional block diagram illustrating a configuration example of the present invention. In FIG. 16, the same or corresponding portions as those in FIG. 6 are denoted by the same or corresponding reference numerals, and description thereof will be omitted.

The unnecessary frame number receiving section 501 is means for receiving an unnecessary frame number from the transmitting side as necessary, and passes the received number to the unnecessary frame erasing section 502.

The unnecessary frame erasing section 502 is means for erasing unnecessary frame data in the frame memory 205 in accordance with an instruction from the transmitting side.

(C-3) Operation at the time of multipoint transmission Here, an operation example according to the third embodiment will be described by taking the case of FIG. 8 used for explaining the operation of the first embodiment as an example.

The target terminal recording unit 403 records, as shown in FIG. 17, which receiving terminal each frame is used as a reference frame. Unwanted frame the target terminal tape b le based selected in unnecessary frame selection unit 404.

For example, since the frame a is referred to by the receiving terminals A and B, it cannot be deleted until the frame e referred to by both the receiving terminals A and B is encoded next. On the other hand, since the frame b is referred to only by the receiving terminal A, the frame c similarly referred to only by the receiving terminal A may be deleted after being encoded.

As described above, when past limited terminals are recorded, frames targeting the same terminal can be sequentially deleted.

However, if the frame b is simply erased, the following problem may occur. For example, it is assumed that a propagation delay with the receiving terminal B is large, and a confirmation signal from the receiving terminal B that has received the frame b arrives shortly before encoding the frame d. At this time, when the frame data is left without being erased, the frame d can be encoded with reference to the frame b as shown in FIG. However, if the frame b is deleted when the frame c is encoded as described above, the receiving terminal A
Although the receiving terminal B and the receiving terminal B have both received the frame b, the reference frame must be returned to the frame a and encoded as shown in FIG.

[0095] Therefore, the unnecessary frame selection unit 404 according to the present embodiment, in order to such a problem does not occur, not only the target terminal tape b le, also with reference to select the unwanted frame reception capability. That is, it is determined whether or not to erase frame b in consideration of the receiving capability of receiving terminal B.

For example, as shown in FIG. 8, when it is estimated that the receiving terminal B receives only once every four frames, the receiving terminal B immediately deletes the frame b. On the other hand, receiving terminal B
If the propagation delay of the confirmation signal is estimated to be large, the frame b is deleted after waiting for the maximum propagation delay time.

(C-4) Effects of the Third Embodiment As described above, in the present embodiment, the same effects as those of the first embodiment can be obtained. In addition, in the third embodiment, The following two effects are obtained as effects unique to the form. One is that the target terminal can successively delete limited reference frames, so that memory capacity can be saved. Another is that the erasure frame is determined in consideration of the reception capability, so that erasure can be performed quickly. That's not too much.

(D) Fourth Embodiment A fourth embodiment will be described. Here,
The fourth embodiment will be described based on the differences between the fourth embodiment and the third embodiment. The difference between the fourth embodiment and the third embodiment is that a means for weighting frame data by a target terminal and a means for erasing frame data from a frame memory according to the weighting are added to the transmitting terminal. is there. In view of the above, the present embodiment has a feature that when a necessary frame must be erased in a system having a small memory capacity or the like, memory erasure can be controlled so that more important frame data remains until the end.

(D-1) Configuration of Moving Picture Coding Apparatus 600 FIG. 20 shows a moving picture coding apparatus 600 according to the fourth embodiment.
FIG. 2 is a functional block diagram illustrating a configuration example of the present invention. In FIG. 20, the same or corresponding parts as those in FIG. 15 are denoted by the same or corresponding reference numerals, and description thereof will be omitted.

Reception ability estimation section 601 has the same function as reception ability estimation section 401 described in the third embodiment.
The estimated receiving capability is transferred to the target terminal limiting unit 602 and the unnecessary frame selecting unit 404, and also to the frame weighting unit 603.

The target terminal limiting unit 602 has the same function as the target terminal limiting unit 402 described in the third embodiment.
The limited target terminal data is used as a reference frame selecting unit 112
And to the frame weighting unit 603 at the same time.

The frame weighting section 603 is a means for calculating a weight indicating the importance of frame data based on the limited number and capabilities of target terminals, and records the calculated weight in the weight recording section 604.

The weight recording unit 604 has a function equivalent to that of the target terminal recording unit 404 described in the third embodiment, and also records the weight of each frame determined by the frame weighting unit 603. At the same time as passing the recorded data to the unnecessary frame selecting unit 404, the data is also passed to the necessary frame erasing unit 605.

When necessary frame data must be deleted from the necessary frame data, the necessary frame deleting unit 605 deletes the frame data in the frame memory 104 according to the weight of the frame recorded in the weight recording unit 604. This is a means for erasing. Also, if the receiving side is also compatible, it passes it to the unnecessary frame number transmitting unit 406.

The configuration of the receiving terminal side, that is, the structure of the moving picture decoding apparatus is the same as that of the moving picture decoding apparatus 500 described in the third embodiment.

(D-2) Operation at the time of multipoint transmission Now, in the moving picture coding apparatus 600 according to the present embodiment,
In a system where the frame memory capacity is not sufficient, for example, when a necessary frame data has to be erased unavoidably, it is characterized by a function capable of efficiently erasing an insignificant frame.

FIG. 21 shows an example of weighting each frame. Note that the receiving terminal A in FIG. 21 receives every frame, the receiving terminal B receives every other frame, and the receiving terminal C receives every second frame.

[0108] At this time, the video encoding apparatus 600, as in the third embodiment, to create the target terminal tape b le based on the received results, operates to erase the reference frame based on this . In FIG. 21, the frame data in the hatched portion has already been erased as unnecessary.

Assuming the above, the moving picture coding apparatus 60
The operation of 0 will be described. The moving picture coding apparatus 600 first weights each receiving terminal. At this time, a weight of 1 is assigned to the receiving terminal having the highest receiving capability, and a terminal having a lower capability is assigned a larger weight according to the capability difference and the frame rate. In the example of FIG. 21, the receiving terminal A is 1, the receiving terminal B is 2, and the receiving end C is 3.

Next, the sum of the weights of the receiving terminals is calculated for each frame. This is the weight of the frame. The weight is greater for frames that target more receiving terminals, and the weight is higher for terminals that have lower capabilities. This is because a terminal having a higher capability can substitute another frame.

At the time of erasing, erasing is performed in ascending order of weight. In the case of the same weight, a temporally old frame is deleted.

When the erasure of a frame is managed by these rules,
Important frames that are older in time are not erased to the end. For example, when only the time is considered without considering the weight, the frames for the three receiving terminals A, B, and C are first deleted in FIG. When such a situation occurs, the receiving terminals A,
When encoding is performed for B and C, reference cannot be made.

On the other hand, according to the rules described in the present embodiment, even if the frames for the receiving terminals A and B or the frames for the receiving terminals A and C are deleted first, the receiving terminal A , B, and C can be substituted for the frames, so that the influence is small.

(D-3) Effects of the Fourth Embodiment As described above, also in the present embodiment, the same effects as those of the first and third embodiments are obtained. The following two are obtained as effects unique to the fourth embodiment. One is to erase from the less important frames,
The scalable coding efficiency of the frame memory capacity can be realized, and the image quality can be provided according to the system.The other is that the most important frames received by all receiving terminals remain until the end. That is, it is difficult to cause a failure in encoding.

(E) Other Embodiments (E-1) In the third and fourth embodiments described above, the system configuration described in the first embodiment is used as a basic configuration. Although described, the present invention is also applicable to the case where the system described in the second embodiment has a basic configuration, and can obtain the same effects as those obtained in the second embodiment.

(E-2) In the third and fourth embodiments, the transmitting terminal transmits a frame number determined to be unnecessary and the receiving terminal deletes the frame number. However, the present invention is not limited to this. It does not do. That is, the receiving terminal may perform the same processing as the transmitting terminal to select an unnecessary frame.

(E-3) Furthermore, in the description so far, an example in which the reference frame is switched for each frame by taking a frame loss as an example has been described, but the present invention is not limited to this. For example, the frame may be divided into a plurality of blocks, it may be determined whether or not each frame is missing, and the reference frame may be switched for each block. As the size of the block is reduced, the ratio of missing or erroneous data is reduced, so that the reference frame update rate is improved and the coding efficiency is also improved.

(E-4) In the description so far, reference frame updating section 113 copies frame data from frame memory 104 to reference frame memory 105 during intra-frame encoding. However, the present invention is not limited to this. It does not do. For example, the intra-frame coding request signal from the Intra / Inter determination unit 106 is passed to the decoding unit 103, and the frame data is directly written from the decoding unit 103 to the reference frame memory 105 during intra-frame coding. Is also good. In this case, the copying process can be omitted.

(E-5) In the above description, the reference frame memory and the frame memory are separately described and described, but the present invention is not limited to this. The frame data may be written on one memory, and the reference frame may be represented by a pointer indicating the memory position. In this case, it required at the time of the reference frame update to copy the reference frame memory F data from the frame memory without need only move the pointer.

(E-6) Furthermore, in the description so far, the operation example has been described as the confirmation signal as a signal notifying that the frame has been received, but the present invention is not limited to this. That is, the confirmation signal may be notified from the receiving terminal to the transmitting terminal when the frame cannot be received. In this case, since the acknowledgment signal is not normally transmitted, the transmitting side can perform encoding one after another without waiting for acknowledgment. Here, when a non-reception signal is received, the reference frame is returned to the position immediately before the non-reception frame and encoded. As a result, the receiving side cannot decode several frames from the frame that could not be received, but immediately switches to the data referencing the frame that could be received, so that decoding becomes possible.

(E-7) In the description of the receiving terminal side, that is, in the description of the moving picture decoding apparatus, a configuration in which the confirmation signal is transmitted after the decoding operation is completed has been described. It is not limited. That is, when the code data is received, whether the data is normal or not may be checked, and if the data is normal, a confirmation signal may be transmitted. This allows
The timing of transmitting the confirmation signal is advanced, and the propagation delay of the confirmation signal is reduced.

(E-8) Further, in the above description, a system configuration in which a plurality of receiving terminals are connected to one transmitting terminal, that is, a transmission operation is described using multipoint transmission as an example. Even in the one-to-one connection, the present invention can be applied to a case where the network is busy and a delay occurs in the reception confirmation and the reference frame cannot be updated every frame.

(E-9) Further, each of the moving picture coding apparatus and the moving picture decoding apparatus is represented by a functional block, but if such a function is provided, even if it is realized by hardware,
It may be realized by software.

(E-10) Further, in the above description, the data to be transmitted is described as moving image data. However, the present invention is not limited to this, and may be audio data or binary data.

(E-11) In the above description, a general transmission system including a moving picture coding apparatus and a moving picture decoding apparatus has been described. Or may be used as a decoding device.

[0126]

As described above, according to the encoding apparatus of the present invention, the reference data can be updated in accordance with the receiving capability of each decoding apparatus, and the transmitted data to be transmitted can be updated. Thus, it is possible to select appropriate reference data. As a result, the coding efficiency can be kept high.

[0127] Further, according to the decoding apparatus of the present invention, was able to achieve an adaptive coding process in the marks Goka device.

Therefore, if a transmission system is constructed using these encoding devices and decoding devices, it is possible to realize a system with higher encoding efficiency than before.

[Brief description of the drawings]

FIG. 1 is a functional block diagram illustrating a video encoding device according to a first embodiment.

FIG. 2 is an explanatory diagram of an encoding method based on Recommendation H.261.

FIG. 3 is an explanatory diagram of an encoding method based on JPEG.

FIG. 4 is an explanatory diagram when a frame is lost during encoding based on Recommendation H.261.

FIG. 5 is an explanatory diagram when a frame is lost during JPEG-based encoding.

FIG. 6 is a functional block diagram showing the video decoding device according to the first embodiment.

FIG. 7 is an explanatory diagram illustrating an example of image encoding immediately after the start of multipoint transmission.

FIG. 8 is an explanatory diagram illustrating an example of image encoding after a predetermined time has elapsed from the start of multipoint transmission.

FIG. 9 is an explanatory diagram when a frame is lost during multipoint transmission.

FIG. 10 is an explanatory diagram in a case where reception timing varies during multipoint transmission.

FIG. 11 is an explanatory diagram showing a comparative example in which reception timing varies during multipoint transmission.

FIG. 12 is a functional block diagram illustrating a video encoding device according to a second embodiment.

FIG. 13 is an explanatory diagram illustrating an example of image encoding during multipoint transmission.

FIG. 14 is an explanatory diagram when a frame is lost during multipoint transmission.

FIG. 15 is a functional block diagram showing a video encoding device according to a third embodiment.

FIG. 16 is a functional block diagram illustrating a video decoding device according to a third embodiment.

FIG. 17 is an explanatory diagram showing a target terminal table.

FIG. 18 is an explanatory diagram illustrating an example of image encoding when a propagation delay is large.

FIG. 19 is an explanatory diagram illustrating an example of image encoding when frame erasure is performed too early.

FIG. 20 is a functional block diagram illustrating a video encoding device according to a fourth embodiment.

FIG. 21 is an explanatory diagram showing an example of weighting a target terminal table.

[Explanation of symbols]

100, 300, 400, 600 ... moving picture coding apparatus, 101 ... moving picture input section, 102 ... coding section, 1
03, 206... Decoding section, 104, 205... Frame memory, 105, 204... Reference frame memory, 1
06 Intra / Inter determination section 107 Code data transmission section 108 Confirmation signal reception section 109 Multipoint reception recording section 110 Reception capability estimation section 111 Target terminal limiting section 112... Reference frame selection unit, 113,
203: Reference frame updating unit, 200, 500: Moving picture decoding device, 201: Coded data limited receiving unit, 2
02: reference frame comparison unit, 207: confirmation signal transmission unit, 208: moving image output unit, 209: refresh signal transmission unit, 210: reception capability change request unit, 30
1, 401, 601: reception capability estimating unit, 302: receiving terminal classifying unit, 303, 402, 602: target terminal limiting unit, 403: target terminal recording unit, 404: unnecessary frame selecting unit, 405: … Unnecessary frame erasing unit, 406…
... Unnecessary frame number transmission unit, 501... Unnecessary frame number reception unit, 502... Unnecessary frame deletion unit, 603.
Frame weighting unit, 604... Weighting recording unit, 60
5. Necessary frame erasing unit.

──────────────────────────────────────────────────続 き Continuation of front page (56) References JP-A-2-302192 (JP, A) JP-A-4-176291 (JP, A) JP-A-1-282981 (JP, A) JP-A-5-302 252511 (JP, A) (58) Field surveyed (Int. Cl. ⁷ , DB name) H04N ^7/ 24-7/68

Claims (11)

(57) [Claims] A data to be transmitted at a certain point in time is predicted
Reference data to be used as a reference at the time of encoding, and transmitted thereafter
Predictively encode the transmitted data at each point in time
Encoder that outputs data and corresponding to this encoder
And a decoding device for decoding incoming encoded data.
In the transmission system that performs the above, the encoding device transmits the encoded data notified from the decoding device.
Of the decoding device based on the reception state at each point in time.
And estimating means for estimating a reception capability based on the estimated result by the estimation means, predictive coding
Reference data that adaptively updates the reference data
Data decoding means, the decoding device, the reception state of the encoded data at each point in time, the
Provided with a reception state notifying unit for notifying the encoding device
A transmission system, characterized in that: 2. The decoding device according to claim 1 , wherein
Request transmission means for transmitting
For example, estimating means of the encoding device, when changing a reception capability request
Re-estimates and accepts request to change reception capability
2. The transmission system according to claim 1, wherein
Stem. 3. The method according to claim 1, wherein the transmitted data at a certain point in time is predicted
Reference data used as a reference at the time of encoding, and predictive encoding of transmitted data at each point in time to be transmitted thereafter, in an encoding device that transmits the encoded data to a decoding device, the notification from the decoding device Estimating means for estimating the receiving capability of the decoding device based on the reception state of each of the coded data at each point in time, based on the estimation result by the estimating means, a reference at the time of predictive encoding, A reference data updating means for adaptively updating reference data to be encoded. 4. The decoding device according to claim 1 , wherein:
When a request for changing the receiving capability is given,
Re-estimate and decide whether to accept
The encoding apparatus according to claim 3, wherein the determination is performed.
Place. 5. A transmission target limiting unit that limits a decoding device that decodes encoded data transmitted at the next point in time from a plurality of decoding devices based on a result of the estimation by the estimation unit. Selecting means for selecting reference data suitable for the decoding device limited by the transmission target limiting means, and adaptively updating the reference data according to a selection result by the selecting means. The encoding device according to claim 3 or 4 . 6. A decoding device having an equivalent receiving capability is classified based on the estimation result by the estimation means, and the encoded data is transmitted at the same timing to decoding devices belonging to the same classification. 6. The encoding apparatus according to claim 3, further comprising a classifying unit for controlling an updating operation in said reference data updating unit. 7. For each transmitted data stored in preparation for the case where it is selected as reference data, the transmitted data is used as a reference data when any decoding device is to be transmitted in the past. A storage unit for recording whether or not the selected transmission device is selected as a storage device, and when there are a plurality of transmission data items having the same combination of decoding devices, the old transmission data item is unnecessary data. and a selecting means for selecting, unnecessary object transmission data is selected, claim 3, characterized in that it comprises an erasing means for erasing from the storage means for storing the data, according to claim 4, claim 5 or The encoding device according to claim 6 . 8. The selecting means converts unnecessary data having a maximum propagation delay time with respect to a decoding device to be transmitted among old transmitted data having the same combination of the decoding devices as unnecessary data. The encoding apparatus according to claim 7 , wherein: 9. For each transmitted data stored in preparation for being selected as reference data, the transmitted data is used as a reference data when any decoding device is to be transmitted in the past. Storage means for recording whether or not the transmission data is selected; weighting means for assigning a greater weight to each transmission data stored in the storage means as the importance is higher ; erasing the transmission data from the storage means Erasing means for erasing transmitted data given a larger weight in the case where it becomes necessary to perform the above-mentioned estimation.
Weighting according to the reception capacity estimated by the means,
Determine the weight of the transmitted data by referring to the transmitted data
Sum of the weights of the decoding devices that have received the encoded data
Claim 3, claim 4, characterized in that the defined value, 請
An encoding device according to claim 5 or claim 6 . 10. A decoding device for sequentially decoding encoded data received via a transmission path, wherein a reception state of the encoded data at each time point is determined by:
A decoding device comprising: a reception state notifying unit that notifies the coding device. 11. The decoding device according to claim 1 , wherein
Request transmission means for transmitting a reception capacity change request
11. The decoding device according to claim 10, wherein: