KR19990001583A - Coding Device, Decoding Device, and Transmission System - Google Patents

Abstract

There was a difference in encoding efficiency due to the reception capability of the decoding apparatus.

The reception capability of the decoding apparatus is estimated based on the reception results at each time point regarding the encoded data notified from the decoding apparatus, and adaptively updated reference data as a reference when predictive encoding is performed based on the estimation result. This makes it possible to update the reference data in accordance with the reception capability of each decoding apparatus.

Description

Coding Device, Decoding Device, and Transmission System

The present invention relates to an image transmission system, an image encoding apparatus, and an image decoding apparatus, and can be applied, for example, to improvement of image quality deterioration due to frame dropping or low speed photography in image transmission.

In recent years, as a video signal transmission system using a network such as a television telephone system, a television conferencing system, or a video on demand (VOD) system has become widespread, the international standardization work of a video encoding method is accompanied by this. This is going on.

The image encoding method can be classified into two types: using intraframe coding and interframe coding together and using only intraframe coding.

Among these, the combination of the intra-frame coding method and the inter-frame coding method is applied to a coding method for moving picture communication / accumulation such as ITU-T Recommendation H. 261 or Moving Picture Experts Group (MPEG). 2 shows an example of frames arranged in chronological order. In ITU-T Recommendation H. 261, intra-frame encoding (I frames a and i) is performed periodically as shown in FIG. 2, but for P frames (inter-frame encoded frames) (b to h and j to) therebetween. By performing inter-frame encoding using the preceding frame as a reference picture, redundancy related to temporal change is excluded. Hereinafter, the frame encoded in the frame is called I frame, and the frame encoded between frames is called P frame.

On the other hand, only intra-frame encoding is used, in-frame encoding is performed for all the frames as shown in Fig. 3 by using a continuous picture coding method such as JPEG (Joint Photographic Coding Experts Group).

Since H. 261 performs inter-frame encoding with reference to the preceding frame, all frames need to be transmitted reliably in order. When data transmission is performed after establishing a line with a partner, such as a telephone line or an ISDN line, there is no problem because data arrives at each other in sequence without dropping out of the way, but LAN and ATM such as Ethernet Since the data is divided into small data units (packets and cells) without being established, the packets may be dropped in the middle, or the order may be changed due to differences in paths.

In general, in order to solve this problem, a protocol in which a sender adds and transmits a number through a packet, changes the order at the receiver, or sends a request for retransmission of an unarrived packet to the sender (TCP: Transmission Control Protocol, etc.) ) To improve network reliability.

However, when the operation of the network is unstable and packets are frequently dropped, delays are accumulated when retransmission processing using the above protocol is not suitable for real-time transmission of moving images. In particular, if new image data can be displayed, it may be better to display new data even if time lapse is taken, rather than retransmitting and displaying old data on purpose.

In addition, broadcast and multicast used when transmitting data to multiple points are devices for transmitting data to multiple points in one packet transmission. In this apparatus, however, if a dropout occurs during packet transmission to one point, retransmission processing similar to the above-described protocol causes the same packet to be received even though the first packet has arrived normally with respect to the network of the other point. Since it is necessary to retransmit, the load on the network increases significantly. Therefore, in broadcast or multicast, it is common to use a protocol (UDP: User Datagram Protocol, etc.) that has not been retransmitted, and the probability of packet dropping is increasing.

In addition, in the case of using a wireless network, the error rate and dropout rate of the data tends to increase not only when the packet is divided and transmitted but also when the circuit is established and transmitted. In particular, even if the receiving side detects an error, when there is an error exceeding the error correction capability, a method of discarding data in any section is used, in order to process other portions as normal, so that the dropout of data is larger than in a wired network.

In addition, there is a problem that the processing speed between the transmitting and receiving terminals is not necessarily identical. For example, when the processing speed of the receiving terminal is slow, when all image frames are decoded and displayed, frame data waiting for processing is accumulated and the delay is increased. Therefore, in such a case, it is necessary for the receiving side to adaptively time-lapse photography. By the way, when a frame is encoded between frames by the conventional technique, there is a problem that the current frame cannot be decoded when there is no decoded data of the previous frame, and low-speed photography is not freely performed.

4 shows an example of frame dropout in the case of H.261. Here, when frame e is dropped out in the middle or processing is delayed, decoding is not possible until the next I frame i arrives, and the frame P (f, g, h) therebetween cannot be decoded.

Thus, conventionally, in order to ensure all frames in a network in which such frames are dropped or time-lapse photography occurs frequently, a method such as JPEG is used in which all frames are transmitted only by intra-frame coding without using inter-frame coding. For example, in the JPEG encoding method shown in Fig. 5, even if frame e is dropped, decoding can be performed normally in the next frame. However, in this case, there is a problem that the redundancy of the temporal change is not excluded because no inter-frame coding is performed, and the coding efficiency is bad and the amount of transmission data is large.

For the above reason, even in an environment where frame dropout or low-speed photography is expected, the P frame can be decoded without waiting for an I-frame to be sent later. In addition, it is possible to suppress a decrease in coding efficiency even during multipoint transmission. There is a demand for the implementation of a coding and decoding device.

In order to solve such a problem, in the present invention, in the encoding apparatus for predicting encoding the transmitted data at each time point to be transmitted thereafter on the basis of the transferred data at some time point, and transmitting the encoded data to the decoding device, It is characterized by the following.

That is, the estimation means for estimating the reception capability of the decoding apparatus in accordance with the reception result of each time point regarding the encoded data notified from the decoding apparatus, and the reference data as a reference when predicting encoding according to the estimation result by the estimation means are adapted. A reference data update means for updating the data is provided.

In the encoding device of the present invention, since the reference data is updated according to the reception capability of each decoding device, it is possible to always select appropriate reference data as compared with the transferred data to be transmitted, and to realize encoding according to the transmission target. have. In addition, since the opportunity to select inappropriate data as reference data can be reduced, the coding efficiency can be maintained high.

According to the present invention, a decoding apparatus for sequentially decoding encoded data received through a transmission path is characterized as follows.

That is, when a change request is made in relation to its own reception capability or when there is a change in the reception capability, a transmitting means is provided for notifying the encoding apparatus that transmits the encoded data of a change in the reception capability that has occurred or has already occurred. do.

Since the decoding apparatus of the present invention can notify the encoding apparatus side of the change of its reception capability, it is possible to realize the adaptive encoding processing in the encoding apparatus.

Then, a transmission system is constructed using these encoding apparatuses and decoding apparatuses to realize a system with high coding efficiency.

1 is a functional block diagram showing a video encoding apparatus according to the first embodiment.

2 is an explanatory diagram of a coding scheme based on Recommendation H. 261;

3 is an explanatory diagram of a coding method based on JPEG.

4 is an explanatory diagram in the case of frame dropout at the time of encoding according to Recommendation H. 261;

5 is an explanatory diagram when a frame dropout occurs at the time of encoding based on JPEG.

Fig. 6 is a functional block diagram showing a moving picture decoding apparatus according to the first embodiment.

7 is an explanatory diagram showing an example of image encoding immediately after the start of multipoint transmission;

8 is an explanatory diagram showing an example of image encoding after a predetermined time has elapsed from the start of multipoint transmission.

9 is an explanatory diagram in the case of frame dropout during multipoint transmission;

10 is an explanatory diagram in the case where there is a change in reception timing at the time of multipoint transmission.

Fig. 11 is an explanatory diagram showing a comparative example when there is a variation in reception timing at the time of multipoint transmission.

Fig. 12 is a functional block diagram showing a moving picture coding apparatus according to the second embodiment.

Fig. 13 is an explanatory diagram showing an example of image encoding in multipoint transmission;

14 is an explanatory diagram in the case of frame dropout during multipoint transmission;

Fig. 15 is a functional block diagram showing a moving picture coding apparatus according to the third embodiment.

Fig. 16 is a functional block diagram showing a moving picture decoding apparatus according to the third embodiment.

17 is an explanatory diagram showing a target terminal table;

18 is an explanatory diagram showing an example of image encoding in the case where the propagation delay is large;

Fig. 19 is an explanatory diagram showing an example of image encoding when frame erasing is too fast.

20 is a functional block diagram showing a video encoding apparatus according to the fourth embodiment.

21 is an explanatory diagram showing a weighting example of a target terminal table.

* Explanation of symbols for main parts of the drawings

100, 300, 400, 600: moving picture encoding apparatus 101: moving picture input unit

102: encoder 103,206: decoder

104,205: frame memory 105,204: reference frame memory

106: Intra / Inter judgment unit 107: code data transmission unit

108: confirmation signal receiving unit 109: multi-point receiving recording unit

110: reception capacity estimation 111: target terminal government

112: reference frame selection 113,203: reference frame update unit

200,500: moving picture decoding apparatus 201: code data limited receiver

202: reference frame comparison unit 207: confirmation signal transmission unit

208: moving image output unit 209: refresh signal transmission unit

210: Request for changing the reception capability 301,401,601: Estimation of the reception capability

302: receiving terminal classification unit 303,402,602: target terminal government

403: target terminal recorder 404: unnecessary frame selection

405: Unnecessary frame eraser 406: Unnecessary frame number transmitter

501: unnecessary frame number receiving unit 502: unnecessary frame erasing unit

603: frame weighting unit 604: weighting recording unit

605: frame elimination required

A first embodiment of the case where the present invention is applied to a multipoint transfer of a moving picture will be described below with reference to the drawings.

(A) (1st embodiment)

The transmission terminal of the transmission system according to the present embodiment is a decoding error presence signal transmitted by each receiving terminal to inform the transmitting terminal that a frame has been normally received (or has an abnormality) when transmitting a multipoint (hereinafter, an acknowledgment signal). Estimates the receiving capability of each receiving terminal, and if there is a difference in receiving capability, limits the target terminal to receive the next frame, and selects and updates the reference frame according to this. It is characterized by a point.

In addition, the receiving terminal of the transmission system according to the present embodiment has a function of not performing decoding processing on the encoded data not targeted for the own terminal, and receives a request for changing the receiving capability of the own terminal from the user. In this case, it is characterized in that a function for transmitting it to the transmitting terminal side is provided.

In other words, the transmission system according to the present embodiment is characterized by having a function of adaptively updating the reference frame in accordance with the capability of the receiving terminal.

Next, the configuration of the transmitting terminal and the receiving terminal constituting the transmission system will be described.

(A-1) Configuration of the video encoding apparatus 100

Fig. 1 is a functional block diagram showing an example of the configuration of a video encoding apparatus 100 as an example of a transmission terminal constituting a transmission system according to the present embodiment.

The moving image input unit 101 is a means for passing moving image data input from a camera or the like to the encoding unit 102 for each frame.

The encoder 102 is a means for encoding the input frame data, and is transmitted to the decoder 103 and the code data transmitter 107. In the encoding, the intra-frame encoding and the inter-frame encoding are switched according to the instructions of the Intra / Inter judging unit 106. Where Intra represents intra-frame encoding and Inter represents inter-frame encoding. In inter-frame encoding, the frame data in the reference frame memory 105 is referred to and encoded.

The decoding unit 103 is a means for decoding the coded data again and writing the decoded frame data to the frame memory 104 simultaneously with the frame number.

The frame memory 104 is a medium for storing every frame data, and data that is no longer needed is erased each time the reference frame is updated by the reference frame updater 113.

The reference frame memory 105 is a medium for storing reference frame data used for inter-frame encoding, and is updated by the reference frame updater 113.

The intra / inter determination unit 106 is a means for determining whether to perform intra-frame encoding or inter-frame encoding. In general, in-frame encoding is performed periodically (for example, every 30 frames) and inter-frame encoding is performed for other periods. However, when a refresh signal is received from the receiving side (the video decoding apparatus 200 described later). Intra-frame encoding is forcibly performed. If it is determined that the frame is encoded, the encoder 102 and the reference frame updater 113 notify that.

The code data transmitter 107 is a means for transmitting the encoded video data to the receiving side (video decoding apparatus 200). The data is multiplied with a flag indicating an intra / inter, a reference frame number, or the like. In addition, if necessary, information of the transmission source, transmission line information, and the like are also multiplexed and transmitted. In the case of multipoint transmission, all data is transmitted to all receiving terminals. When limited to some terminals from the target terminal limiting unit 111, the terminal may be transmitted only to the terminal. In addition, the target terminal information may be multiplexed and transmitted.

The acknowledgment signal receiving unit 108 is a means for receiving an acknowledgment signal of which frame has been decoded by the receiving side (the video decoding apparatus 200), and passes the confirmed frame number to the reference frame updating unit 113. When multipoint transfer is being performed, the data is transferred to the multi-point reception recording unit 109.

The multi-point reception recording section 109 is a means for recording, by each terminal, the received reception records such as which frames each reception terminal has received so far, and the propagation delay until transmitting the encoded data and receiving the confirmation signal. The received data is passed to the reception capability estimation 110, the target terminal 111, and the reference frame selecting unit 112. In addition, when the reference frame selection unit 112 selects a frame received by all receiving terminals as a reference frame, the old data before that may be erased or may be stored for a predetermined period.

The reception capability estimator 110 estimates the reception capability (frame rate or propagation delay of the network) by calculating the probability of dropping the average frame of each receiving terminal based on the past reception records recorded in the multi-point reception recorder 109. Sudan, and handed over to the government 111 the target terminal. The capability estimation may be performed only once after a certain time of communication start, or may be performed every fixed time when there is a change in the reception capability or the state of the network. When the reception capability change request is received from the reception side (video decoding apparatus 200), the reception capability estimation is performed according to the instruction. However, in the case where it is estimated that the request for changing the reception capability from the reception side is difficult to realize, it may be notified to the reception side so as not to change the capability.

The target terminal limiting unit 111 is a means for limiting a terminal that receives the next frame from the reception records recorded in the multi-point reception recording unit 109 based on the reception capability of each terminal estimated by the reception capability estimating unit 110. In general, all terminals are subject to reception, but clearly there is a difference in reception capability, and when there is a terminal that is expected not to receive the next frame, the terminal is restricted and the limited terminal data is referred to the reference frame selecting unit 112. The data is transmitted to the code data transmitting unit 107 at the same time.

The reference frame selecting unit 112 is a means for selecting the frame most commonly received as the reference frame based on the reception result of the terminal defined in the target terminal limiting unit 111, and passes the selected frame number to the reference frame updating unit 13.

The reference frame updating unit 113 is a means for updating the reference frame in accordance with the instructions from the Intra / Inter determination unit 106, the confirmation signal receiving unit 108, and the reference frame selection unit 112.

When the intra-frame encoding instruction is received from the intra / inter determination unit 106, the data of the current frame is copied from the frame memory 104 to the reference frame memory 105, and all frame data of the frame memory 104 is erased. When the frame number is received from the confirmation signal receiving unit 108, the frame data of the number is copied from the frame memory 104 to the reference frame memory 105, and the frame data preceding the number is erased from the frame memory 104. In the case of multipoint transmission, the frame data of the number is copied from the frame memory 104 to the reference frame memory 105 in accordance with an instruction from the reference frame selecting unit 112. At this time, only when the target terminal is not limited, the frame data before the number is erased from the frame memory 104.

(A-2) Configuration of the moving picture decoding apparatus 200

6 is a functional block diagram showing an example of the configuration of a moving picture decoding apparatus 200 as an example of a receiving terminal constituting a transmission system according to the present embodiment.

The code data limited reception unit 201 is a means for receiving the encoded video data from the transmitting side (video encoding apparatus 100). A flag indicating a coded data and an intra / inter multiplexed to the data, a reference frame number, and the like are passed to the decoding unit 206, and the transmission source information, etc., which are multiplexed as necessary, is also passed to the decoding unit 206. In addition, the reference frame number is passed to the reference frame comparison unit 202. When the target terminal information is multiplexed during the multipoint transmission, the information is also passed to the reference frame comparison unit 202. If the own terminal is not the target, the received encoded data is discarded, and no decoding process is performed for the frame.

The reference frame comparison unit 202 compares the received reference frame number with the reference frame numbers of its own terminals stored in the reference frame memory 204, and notifies the reference frame update unit 203 of the reference frame update request when the reference frame numbers are different. The new reference frame number is passed. If there is target terminal information, it is also passed on.

Upon receiving an update request from the reference frame comparator 202, the reference frame updater 203 reads data of a new reference frame number from the frame memory 205, and simultaneously writes and updates the frame number to the reference frame memory 204. At this time, when the target terminal is not limited, the old frame data on the frame memory is erased.

The reference frame memory 204 is a medium for storing reference frame data used for interframe decoding, and is updated by the reference frame updating unit 203 and the decoding unit 206.

The frame memory 205 is a medium for storing decoded frame data.

The decoding unit 206 is a means for decoding the input coded data, and sends it to the decoded data moving image output unit 208. When decoding, in the case of an I frame, the recording reference frame is updated in the reference frame memory 204 at the same time as the decoded data. In the case of the P frame, the data is decoded with reference to the data in the reference frame memory 205, and the decoded data is recorded in the frame memory 205 simultaneously with the frame number. If the decoding is successful, the frame number is passed to the confirmation signal transmitter 207.

The acknowledgment signal transmitting unit 207 is means for transmitting an acknowledgment signal indicating which frame has been decoded to the transmitting side (video encoding apparatus 100), and also transmits the frame number received from the decoding unit 206 together.

The moving image output unit 208 is a means for outputting decoded data received from the decoding unit 206 to a monitor or the like.

The refresh signal transmitter 209 is a means for transmitting a refresh signal to the transmitting side (the video encoding apparatus 100) in response to a refresh request from the user.

The reception capability change request means 210 is a means for sending a reception capability change request to the transmission side (video encoding apparatus 100) when a change request is made from the user with respect to the reception capability of the reception terminal estimated by the transmission side. Here, when the request from the user exceeds the capability of the receiving terminal, the user may be notified of the purpose and the change request may not be sent.

(A-3) Operation in multipoint transmission

Next, operation examples of the video encoding apparatus 100 and the video decoding apparatus 200 during multipoint transmission will be described. Here, the operation states of these two devices 100 and 200 are divided into two periods, one for a period from the start of communication and a period after that. In this explanation, the number of moving picture decoding apparatuses 200 connected to the moving picture coding apparatus 100 is assumed to be two, and each is represented as a receiving terminal A and B.

First, an example of operation between a period of time from the start of communication will be described. Also, the video encoding apparatus 100 at this time has not yet grasped the capability of the receiving terminal connected to the same apparatus. At this time, the moving picture encoding apparatus 100 notifies that the image is normally received for both the receiving terminals A and B based on the confirmation signal (or refresh signal) transmitted for each of the receiving terminals A and B. Only when the reference frame is updated.

That is, as shown in Fig. 7, the reference frame is updated only for the frames a, e, and i, and for each of the other frames b, c, d, f, g, h, j, k at each time point. The encoding is performed based on a frame stored as a reference frame. In the example of Fig. 7, the frames b, c and d are based on the frame a, and the frames f, g and h are based on the frame e. , And so on.

Therefore, in this period, the reference frame is updated in accordance with the reception terminal B having a relatively low reception capability, and encoding of each frame is performed.

This eliminates the need for intraframe coding of all frames in the event of dropping out of frames, and enables decoding of P frames without waiting for subsequent I frames, resulting in higher transmission than conventional systems. Efficiency can be obtained. However, when the timing at which both receiving terminals A and B can normally receive at the same time is significantly shifted, or when the receiving capability of both receiving terminals is large, it is expected that the number of update of the reference frame is reduced. For this reason, the coding efficiency may not always improve as expected when coding with this technique.

Therefore, what is used is the past reception accumulated in a period of time from the start of this communication, and the frame encoding method is modified by estimating the reception capability of each receiving terminal A and B based on this performance. do.

The reception capability estimation unit 110 is used to estimate this reception capability. In the case of FIG. 7, the reception capability estimator 110 can receive almost every frame with respect to the receiving terminal A, whereas the receiving terminal B has only the ability to receive only once every 4 frames with respect to the receiving terminal B. It is estimated.

Based on this result, the target terminal limiting unit 111 determines that the reception terminal B cannot receive up to the frame i of the four frame lines after receiving the frame e, for example, and the frame f therebetween. , g, h limit the target terminal to encode only terminal A.

That is, since the receiving terminal A receives every frame, as shown in FIG. 8, the updating of the reference frame is also performed every frame, and only when the terminal i encodes the frame i determined that both terminals can receive it at the same time. It operates to return the reference frame to frame e which is commonly received.

As a result, the encoding efficiency of the receiving terminal A is lower than that of the one-to-one communication, but not as bad as that of the receiving terminal B. Therefore, the encoding efficiency is improved more than a period of time from the start of communication.

9 shows a case where frame dropping occurs in the middle of this encoding. If a dropout occurs (frames c, e and I in the figure), the acknowledgment signal receiver 108 receiving the acknowledgment signal detects this and causes the next frame to be encoded without updating the reference frame. As a result, the deterioration of the image quality is unlikely to occur even for the dropped frame.

10 shows a case where the reception timings of both receiving terminals do not match. In this case, since the other terminal is limited to each frame, update of a reference frame suitable for each receiving terminal is realized, and only a frame that both terminals can receive in common is used as a reference frame (Fig. 11). In comparison to the above, it is possible to realize an improvement in the coding efficiency.

(A-4) Effect of 1st Embodiment

According to the above configuration, since the next frame is encoded by controlling the update of the reference frame in accordance with the reception result of each receiving terminal, the receiving terminal can decode all the frames without waiting for the next I frame to be transmitted. The efficiency can be improved.

Further, by providing a function for estimating the reception capability of each receiving terminal in the transmitting terminal, even if the number of receiving terminals increases, encoding of a frame suitable for each receiving terminal and transmission of data can be realized.

This is effective in multipoint transmission, and even if a receiving terminal having a different receiving capability is connected to one line, the influence of the receiving terminal having low capability is small, and high coding efficiency can be maintained for a receiving terminal having high capability.

In addition, even when the reception timing between a plurality of receiving terminals is shifted during multipoint transmission, the reference frame suitable for each terminal can be updated, and high coding efficiency can be maintained.

(B) (2nd embodiment)

Subsequently, a second embodiment will be described. The difference between the second embodiment and the first embodiment is a means for grouping receiving terminals having the same reception capability with respect to a plurality of receiving terminals receiving images from the transmitting terminal on the transmitting terminal side; For the receiving terminals classified into the same group, a means for restricting the transmission of the frames at the same timing is added, and the load of the transmission processing is reduced.

(B-1) Configuration of the moving picture coding apparatus 300

12 is a functional block diagram showing an example of the configuration of a video encoding apparatus 300 according to the second embodiment. In Fig. 12, the same or corresponding parts as those in Fig. 1 are denoted by the same or corresponding reference numerals and the description thereof will be omitted.

The reception capability estimating unit 301 has the same function as the reception capability estimating unit 110 described in the first embodiment, and passes the estimation result to the target terminal limiting unit 303 and also to the receiving terminal classifying unit 302.

The receiving terminal classifying unit 302 is a means for grouping each receiving terminal by capability based on the receiving capability estimated by the receiving capability estimating unit 301, and passes the classification result to the target terminal limiting unit 303. The target terminal limiting unit 303 has the same function as the target terminal limiting unit 111 described in the first embodiment and is always the same timing for the receiving terminals classified into the same group based on the classification result of the receiving terminal classifying unit 302. It is limited to. As a result, transmission of the encoded data is guaranteed at the same time for the receiving terminals classified into the same group.

The receiving terminal side, that is, the structure of the moving picture decoding apparatus is the same as that of the first embodiment.

(B-2) Operation in multipoint transmission

The decrease in the coding efficiency due to the mismatching of the reception timings of the two receiving terminals is also avoided by the video encoding apparatus 100 according to the first embodiment. However, in the case of this video encoding apparatus 100, it is alternately performed at the same timing. Waste is included because encoding and transmission are repeated.

Therefore, in the moving picture coding apparatus 300 according to the second embodiment, as shown in Fig. 13, receiving terminals A and B having the same capability are grouped and transmitted at the same timing.

In this way, useless processing on the transmitting terminal side can be omitted, and processing efficiency can be improved.

In this example, Fig. 14 shows a case where a dropout of the frame occurs. Even in this case, if a dropout occurs, the frame is encoded without updating the reference frame, so that deterioration of image quality due to dropout does not occur much. In addition, in this embodiment, receiving terminals classified into the same group are treated as exactly the same thing.

(B-3) Effects of the Second Embodiment

As described above, also in the case of the present embodiment, the same effects as in the case of the first embodiment can be obtained, and in addition, the following two kinds can be obtained as the original effects of the second embodiment. First, since receiving terminals with the same reception capability are treated in the same state, there is no fear that the timing of encoding and transmission processing may be shifted, and there is no waste of processing. Since it treats as a state, in the target terminal limitation process, the combination of the terminal which generate | occur | produces is drastically reduced, and the process can be simplified.

(C) (third embodiment)

Again, a third embodiment will be described. The difference between the third embodiment and the first embodiment is a means for recording, on the transmitting terminal side, which receiving terminal was previously selected as the reference frame with respect to the frame data stored by the transmitting terminal. And means for selecting old frames that are a combination of the receiving terminals as unnecessary frames, means for erasing the selected frame data from the frame memory, and means for transmitting unnecessary frame numbers to the receiving side. And means for receiving unnecessary frame numbers and means for erasing unnecessary frames on the receiving terminal side. In view of the foregoing, the present embodiment is characterized in that it is possible to adaptively erase unnecessary pieces of frame data which are candidates for the reference frame according to the capability of the receiving terminal.

(C-1) Configuration of the moving picture coding apparatus 400

Fig. 15 is a functional block diagram showing an example of the configuration of a moving picture coding apparatus 400 according to the third embodiment. In Fig. 15, the same or corresponding parts as those in Fig. 1 are denoted by the same or corresponding reference numerals, and the description thereof is omitted.

The reception capability estimation unit 401 has the same function as the reception capability estimation unit 110 described in the first embodiment, passes the estimated reception capability to the target terminal limiting unit 402, and also passes unnecessary frame selection unit 404.

The target terminal limiting unit 402 has the same function as the target terminal limiting unit 111 described in the first embodiment, passes the limited target terminal data to the reference frame selecting unit 112 and also the target terminal recording unit 403.

The target terminal recording unit 403 is a means for recording to which terminal the frame data recorded in the frame memory 104 has been selected as a reference frame in the past, and passes the recorded data to the unnecessary frame selection unit 404. In addition, a dedicated table may be prepared for recording of the target terminal or may be added to the header information of the frame memory 104.

The unnecessary frame selection unit 404 is a means for selecting an unnecessary frame that is not selected in the future from the frame data recorded in accordance with the reception capability, and passes the selected frame number to the unnecessary frame elimination unit 405. In addition, if the receiving side also supports, it is passed to the unnecessary frame number transmitter 406 as well.

The unnecessary frame erasing unit 405 is means for erasing unnecessary frame data on the frame memory 104 in accordance with the instruction of the unnecessary frame selection unit 404.

The unnecessary frame number transmitter 406 is a means for transmitting an unnecessary frame number to the receiving side as needed.

(C-2) Configuration of Moving Picture Decoding Device 500

FIG. 16 is a functional block diagram showing a configuration example of a moving picture decoding apparatus 500 according to the third embodiment. In Fig. 16, the same or corresponding parts as those in Fig. 6 are denoted by the same or corresponding reference numerals and the description thereof is omitted. The unnecessary frame number receiving unit 501 is a means for transmitting an unnecessary frame number from the transmitting side as necessary, and passes the received number to the unnecessary frame erasing unit 502.

The unnecessary frame erasing unit 502 is means for erasing unnecessary frame data on the frame memory 205 in accordance with an instruction from the transmitting side.

(C-3) Operation at Multipoint Transmission

Here, the example of operation concerning 3rd Embodiment is demonstrated using the case of FIG. 8 used for description of operation of 1st Embodiment as an example.

In the target terminal recording section 403, as shown in Fig. 17, which receiving terminals are used for which receiving terminals are recorded in the reference frame. Based on this target terminal table, unnecessary frames are selected by unnecessary frame selection unit 404.

For example, since frame a is referred to receiving terminals A and B, it cannot be erased until the next frame e that is referenced to both receiving terminal A and receiving terminal B is encoded. On the other hand, since the frame b is only referred to the receiving terminal A, the frame c similarly referred to only the receiving terminal A may be erased after encoding.

In this way, if the past limited terminal is recorded, the frames targeted for the same terminal can be sequentially erased.

However, simply erasing frame b may cause the following problems. For example, suppose that the propagation delay between the receiving terminal B is large and the confirmation signal at the receiving terminal B, which has received the frame b, arrives just before the encoding process of the frame d. At this time, if the frame data is left without erasing, the frame d can be encoded with reference to the frame b as shown in FIG. However, when the frame b is erased at the time when the frame c is encoded as described above, although the receiving terminal A and the receiving terminal B receive the frame b at the same time, as shown in FIG. Must be coded back.

Therefore, in order to prevent such a problem from occurring in the unnecessary frame selection unit 404 according to the present embodiment, unnecessary frames are selected by referring not only to the target terminal table but also to the reception capability. That is, it is determined whether or not to erase the frame b in consideration of the reception capability of the reception terminal B.

For example, as shown in FIG. 8, when it is estimated that the receiving terminal B receives only once in four frames, the receiving terminal B immediately erases the frame b. On the other hand, when it is estimated that the propagation delay of the confirmation signal of the reception terminal B is large, the maximum propagation delay time is waited and the frame b is erased.

(C-4) Effects of the Third Embodiment

As described above, also in the case of the present embodiment, the same effects as in the case of the first embodiment can be obtained, and in addition, the following two effects can be obtained as the original effects of the third embodiment. One is that the target terminal can sequentially erase the limited reference frames, thereby saving memory capacity. The other is that erasing is not too fast because the erase frame is determined in consideration of the reception capability. will be.

(D) (fourth embodiment)

The fourth embodiment will be described again. In addition, the 4th Embodiment is demonstrated from the point of difference with 4th Embodiment and 3rd Embodiment here. The difference between the fourth embodiment and the third embodiment is that a means for weighting the frame data by the target terminal on the transmitting terminal side and a means for erasing the frame data from the frame memory in accordance with the weighting. In view of the above, in the present embodiment, when the required frame must be erased by a system having a small memory capacity or the like, memory erasing can be controlled so as to remain as last as important frame data.

(D-1) Configuration of the moving picture coding apparatus 600

20 is a functional block diagram showing an example of the configuration of a video encoding apparatus 600 according to the fourth embodiment. In Fig. 20, the same or corresponding parts as those in Fig. 15 are denoted by the same or corresponding reference numerals, and the description thereof is omitted.

The reception capability estimating unit 601 has a function equivalent to that of the reception capability estimating unit 401 described in the third embodiment, passes the estimated reception capability to the target terminal limiting unit 602 and the unnecessary frame selection unit 404, and the frame weighting unit. Pass it to 603.

The target terminal limiting unit 602 has the same function as the target terminal limiting unit 402 described in the third embodiment, and passes the limited target terminal data to the reference frame selecting unit 112 and the weight recording unit 604, and also to the frame weighting unit 603. Pass it over.

The frame weighting unit 603 is a means for calculating the specific gravity indicating the importance of the frame data by the limited number of target terminals and the capability, and records the calculated specific gravity in the weighting recording unit 60.

The weight recording unit 604 has a function equivalent to that of the target terminal recording unit 404 described in the third embodiment, and also means for recording the specific gravity of each frame determined by the frame weighting unit 603. The recorded data is passed to the unnecessary frame selection unit 404 and also passed to the required frame erasure unit 605.

The necessary frame erasing unit 605 is a means for erasing the frame data on the frame memory 104 in accordance with the specific gravity of the frame recorded in the weight recording unit 604 when it is necessary to erase any of the necessary frame data. If the receiving side also supports, it is also passed to the unnecessary frame number transmitter 406.

The configuration of the receiving terminal, that is, the video decoding apparatus, is the same as that of the video decoding apparatus 16 described in the third embodiment.

(D-2) Operation at Multipoint Transmission

In the moving picture encoding apparatus 600 according to the present embodiment, when the frame data necessary for a system, such as a frame memory capacity is insufficient, must be erased inevitably, a function capable of efficiently erasing from an insignificant frame is characterized. Since it exists, it demonstrates centering on this point.

21 shows a weighting example of each frame. Receiving is performed for each frame in the receiving terminal A in FIG. 21, receiving is performed in one receiving frame B in the receiving terminal B, and receiving in two receiving frames is received in the receiving terminal C. FIG.

At this time, the moving picture encoding apparatus 600 is similar to the case of the third embodiment, and operates to create a target terminal table in accordance with the reception results and thereby erase the reference frame. In Fig. 21, the frame data of the oblique portion is already deleted because it is unnecessary. On the premise of the above, the operation of the video encoding apparatus 600 will be described. The moving picture encoding apparatus 600 first weights each receiving terminal. At this time, the specific gravity 1 is applied to the receiving terminal having the highest reception capability, and the greater the weight of the terminal having the lower capability due to the difference in capability or the frame rate. In the example of Fig. 21, the receiving terminal A is 1, the receiving terminal B is 2, and the receiving terminal C is 3.

Next, the sum of the specific gravity of the receiving terminal in each frame is calculated. This is the specific gravity of the frame. Frames that target a large number of receiving terminals have a greater weight, and targets a terminal with low capabilities have a larger weight. This is because a terminal with higher capability can substitute for another frame.

At the time of erasing, erasing is performed sequentially from the frame having the small specific gravity. If the weight is the same, the frames that are older in time are erased.

By managing the erasure of frames by this rule, an important frame is not erased to the end even if it is old in time. For example, in the case where only time is taken into account without considering specific gravity, the frames for three receiving terminals A, B, and C are deleted for the first time in FIG. If such a situation occurs, it cannot be referred to later when encoding is performed on the receiving terminals A, B, and C.

On the other hand, according to the rules shown in this embodiment, even if the frame for receiving terminals A and B or the frame for receiving terminals A and C are first erased, the receiving terminals A, B and C are targeted. Less impact because it can be used as a frame.

(D-3) Effect of 4th Embodiment

As mentioned above, also in this embodiment, the same effect as the case of 1st, 3rd embodiment can be acquired, and in addition, the following two can be obtained as an effect of the 4th embodiment original. One of them is erasing from a frame of low importance so that a scalable coding efficiency can be realized in the frame memory capacity, so that the image quality can be provided according to the system. The most important frame you're doing remains until the end, so it's hard to break the coding.

(E) another embodiment

(El) In addition, in the above-described third and fourth embodiments, the differences between the system configurations described in the first embodiment are described as basic configurations, respectively, and the differences thereof are explained. Even when it is set as the structure, application is possible and the effect similar to what was obtained by 2nd Embodiment can be acquired, respectively.

(E # 2) In addition, in the third and fourth embodiments, a configuration is shown in which the frame number determined to be unnecessary on the transmitting terminal side and then erased on the receiving terminal side is not limited thereto. In other words, the receiving terminal side may perform the same processing as the transmitting terminal side to select unnecessary frames.

(E-3) In the above description, the frame dropping is taken as an example, and an example of changing the reference frame for each frame is shown, but the present invention is not limited thereto. For example, the frame may be divided into a plurality of blocks to determine whether each block is dropped, and the reference frame may be changed for each block. The smaller the block is, the lower the rate of dropping or mistake is made. Thus, the update rate of the reference frame is improved and the coding efficiency is also improved. (E-4) In the above description, the reference frame updater 113 copies the frame data from the frame memory 104 to the reference frame memory 105 during intra-frame encoding. However, the present invention is not limited to this. For example, the intra-frame encoding request signal from the Intra / Inter judging unit 106 may be passed to the decoding unit 103, and the frame data may be directly written from the decoding unit 103 to the reference frame memory 105 during intra-frame encoding. In this case, the copying process can be omitted.

(E # 5) In the above description, the reference frame memory and the frame memory are separately described and described, but the present invention is not limited thereto. Frame data may be recorded on one memory, and the reference frame may be displayed by a pointer or the like indicating a memory location. In this case, data need not be copied from the frame memory to the reference frame memory at the time of updating the reference frame, and only the pointer needs to be moved.

(E-6) In addition, although the confirmation signal showed the operation example as a signal which notifies that the frame was received, it is not limited to this. That is, the confirmation signal may be notified from the receiving terminal side to the transmitting terminal side when the frame cannot be received. In this case, since the acknowledgment signal is not normally transmitted, the transmitting side can perform encoding sequentially without waiting for confirmation. When the non-received signal is received, the reference frame is returned before one of the non-received frames to be encoded. This makes it impossible to decode a number of frames from a frame that could not be received on the receiving side. However, since a frame is immediately converted into data referring to the received frame, decoding becomes possible.

(E # 7) In addition, although the structure of transmitting the confirmation signal after the decoding operation has been described in the description of the receiving terminal side, that is, the video decoding apparatus, the present invention is not limited to this. In other words, when the code data is received, it may be checked whether or not the data is normal, and if it is normal, a confirmation signal may be transmitted. This speeds up the timing of transmitting the confirmation signal and reduces the propagation delay of the confirmation signal.

(E-8) In the above description, the transmission operation has been described using a system configuration in which a plurality of receiving terminals are connected to one transmitting terminal, i.e., multipoint transmission. It is also applicable to cases where the delay occurs in acknowledgment and the reference frame cannot be updated every frame.

(E-9) In addition, although each of the moving picture coding apparatus and the moving picture decoding apparatus is indicated by a functional block, it may be realized by hardware or by software if such a function is provided.

(E # 10) In addition, although the to-be-transmitted data was demonstrated as moving picture data in the previous description, it is not limited to this, It does not matter whether it is audio data or binary data.

(E-11) In the above description, the general description of the transmission system composed of the moving picture coding apparatus and the moving picture decoding apparatus has been given, but of course, it may be used as an encoding device of a storage medium system or may be used as a decoding device. to be.

As described above, according to the encoding apparatus of the present invention, the receiving capability of the decoding apparatus is estimated based on the reception result of each time point regarding the encoded data notified from the decoding apparatus, and the prediction encoding is performed based on the estimation result. Since the reference data is adaptively updated, the reference data can be updated according to the reception capability of each decoding apparatus, so that the appropriate reference data can be selected as compared with the transferred data to be transmitted. As a result, the coding efficiency can be maintained high.

In addition, according to the decoding apparatus of the present invention, when there is a change request for its own reception capability or when there is a change in the reception capability, the decoding apparatus that has been generated or has already occurred to itself with respect to an encoding apparatus that transmits coded data. Since the change can be notified, the adaptive encoding processing in the encoding device can be realized.

Therefore, when a transmission system is constructed using these encoding apparatuses and decoding apparatuses, a system having a higher coding efficiency than the conventional one can be realized.

Claims (9)

A coding apparatus for predicting encoding data to be transmitted at each time point to be transmitted thereafter, based on the data to be transmitted at a certain time point, and transmitting the encoded data to the decoding device, Estimation means for estimating a reception capability of the decoding apparatus in accordance with a reception result of each time point of the encoded data notified from the decoding apparatus; And reference data updating means for adaptively updating reference data as a reference when predicting encoding according to the estimation result by said estimation means. The method of claim 1, The reference data updating means is Transmission object limiting means for limiting, among a plurality of decoding apparatuses, which decode the coded data transmitted at a next time point in accordance with the estimation result by said estimation means; And selecting means for selecting reference data suitable for the decoding apparatus defined by the transmission subject limitation means, And the reference data is adaptively updated in accordance with the selection result by the selection means. The method according to claim 1 or 2, Classifying decoding apparatuses having equivalent reception according to the estimation result by the estimating means, and controlling the updating operation in the reference data updating means so that the encoded data is transmitted at the same timing to the decoding apparatuses belonging to the same classification. And a classification means. The method according to claim 1, 2 or 3, Storage means for recording which to-be-transmitted data has been selected as reference data in the past when the to-be-transmitted data is selected as reference data for each of the to-be-transmitted data stored in preparation for being selected to the reference data; Selecting means for searching for means and selecting old to-be-transmitted data as unnecessary data, when there are a plurality of to-be-transmitted data having the same combination of decoding apparatuses; And erasing means for erasing the selected unnecessary to-be-transmitted data by storage means for holding the data. The method of claim 4, wherein And said selecting means selects, from the old to-be-transmitted data of the same combination of said decoding apparatuses, the data which passed the maximum propagation delay time with respect to the decoding apparatus which is a transmission object as unnecessary data. The method according to claim 1, 2 or 3, Storage means for recording, for each to-be-transmitted data stored in preparation for being selected as reference data, which decoding apparatus has been selected as reference data when a target apparatus to be transmitted has been selected as a transfer target; Weighting means for assigning a weight to each transferred data stored in the storage means according to importance; And an erasing means for erasing the transmitted data from said storage means in order of a large proportion of the transferred data. The method of claim 6, The weighting means weights each of the decoding apparatuses according to the reception capability estimated by the estimating means, and the weighting of the decoding apparatus that receives the encoded data using the weight of each transmitted data as reference data. A coding apparatus, characterized in that the sum value of. A decoding apparatus that sequentially decodes encoded data received through a transmission path, When there is a change request for its own reception capability or when there is a change in the reception capability, it is provided with a transmitting means for notifying the encoding apparatus which transmits the encoded data to notify the change of the reception capability that has occurred or has already occurred. Decoding apparatus, characterized in that. A transmission system comprising the encoding device according to any one of claims 1 to 7, and the decoding device according to claim 8.