WO2023171376A1

WO2023171376A1 - Encoding device and encoding method

Info

Publication number: WO2023171376A1
Application number: PCT/JP2023/006371
Authority: WO
Inventors: 剛士木村
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2022-03-10
Filing date: 2023-02-22
Publication date: 2023-09-14

Abstract

The present disclosure pertains to an encoding device and encoding method configured so as to be able to save frame memory and suppress image degradation. The encoding device comprises an encoding unit, that, on the basis of externally-supplied reference surface control information, stores a plurality of images encoded prior to a prescribed image from a dynamic image that is to be encoded, said plurality of images being stored as reference surface candidates in a storage unit, and the encoding unit also setting any of the plurality of images constituting the reference surface candidates as a reference picture, and encoding the prescribed image. The present technology can be applied to, for example, an image processing system for distributing a dynamic image at low latency, etc.

Description

Encoding device and encoding method

The present disclosure relates to an encoding device and an encoding method, and particularly relates to an encoding device and an encoding method that can save frame memory and suppress image quality deterioration.

The AVC (Advanced Video Coding) method, HEVC (High Efficiency Video Coding) method, etc. are widely used as compression encoding methods used to deliver video images in real time. In delivering video images with low delay, encoding is performed without image rearrangement, and in this case, the temporal distance between the image to be encoded and the reference picture (hereinafter referred to as picture distance) is It is common to use a nearby previous image as a reference picture. In order to achieve low-delay delivery, it is necessary to reduce the size of the encoder and decoder buffers, and for this purpose it is necessary to perform encoding so that the amount of code generated for each image is smoothed. In some cases, the amount of generated code is smoothed at the slice level in order to further reduce the delay.

If an error occurs in the transmission path in such a low-latency streaming environment, it is necessary to restore normality, but as mentioned above, the amount of code generated is smoothed and the buffer for transmission and reception is minimized, so the amount of code is reduced. It is not possible to restore the image quality by inserting a large I (Intra) picture. If an I-picture that is small enough to fit in the buffer is inserted, the image quality will deteriorate at the point of recovery from the error. Furthermore, since images continue to refer to images whose image quality has deteriorated, the image quality continues to deteriorate for a while. In a one-to-many distribution system, if an error occurs in a distribution channel, an I-picture is inserted to restore the system, but the image quality deteriorates even in channels where no errors occur.

As a method for recovering from an error to avoid image quality deterioration, it is conceivable to switch the reference picture to refer to an image that skips over the image in which the error occurred, instead of inserting an I picture (see, for example, Patent Document 1).

Special table 2015-515768 publication

However, in order to recover correctly by referring to the image that skipped over the image where the error occurred, the frame memory of the encoder and decoder must be maintained because reference picture candidates that jump over the image where the error occurred must be maintained. I end up consuming a lot.

The present disclosure has been made in view of this situation, and aims to save frame memory and suppress image quality deterioration.

The encoding device according to the first aspect of the present disclosure uses a plurality of images encoded before a predetermined image of a moving image to be encoded as reference plane candidates based on reference plane control information supplied from the outside. and an encoding unit that encodes the predetermined image by storing it in a storage unit as a reference plane candidate and using any one of the plurality of images that are the reference plane candidates as a reference picture.

In the encoding method according to the first aspect of the present disclosure, an encoding device converts a plurality of images encoded before a predetermined image of a moving image to be encoded into externally supplied reference plane control information. The predetermined image is then stored as a reference plane candidate in the storage unit based on the reference plane candidate, and the predetermined image is encoded using any one of the plurality of images that are the reference plane candidates as a reference picture.

In the first aspect of the present disclosure, a plurality of images encoded before a predetermined image of a moving image to be encoded are stored as reference plane candidates based on reference plane control information supplied from the outside. is memorized. The predetermined image is encoded using any one of the plurality of images that are the reference plane candidates as a reference picture.

The encoding device according to the second aspect of the present disclosure performs encoding before a predetermined image of the video to be encoded, based on transmission delay information from a transmission destination of a bitstream encoding the video. The control unit includes a control unit that determines reference plane control information for controlling which of the plurality of images is stored in the storage unit as a reference plane candidate, and supplies the information to an encoding unit that encodes the predetermined image.

In the encoding method according to the second aspect of the present disclosure, the encoding device selects a predetermined image of the video to be encoded based on transmission delay information from a destination of a bitstream that encodes the video. Reference plane control information for controlling which of the plurality of previously encoded images is to be stored in the storage unit as a reference plane candidate is determined and supplied to the encoding unit that encodes the predetermined image.

In a second aspect of the present disclosure, based on transmission delay information from a transmission destination of a bitstream in which a moving image is encoded, a plurality of bitstreams encoded before a predetermined image of the moving image to be encoded are transmitted. Reference plane control information that controls which of the images is to be stored in the storage unit as a reference plane candidate is determined and supplied to an encoding unit that encodes the predetermined image.

Note that the encoding devices according to the first and second aspects of the present disclosure can be realized by causing a computer to execute a program. A program to be executed by a computer can be provided by being transmitted via a transmission medium or recorded on a recording medium.

The encoding device may be an independent device or may be an internal block forming one device.

FIG. 2 is a diagram illustrating a problem that the image processing system of the present disclosure attempts to solve. FIG. 2 is a diagram illustrating a problem that the image processing system of the present disclosure attempts to solve. FIG. 2 is a diagram illustrating a problem that the image processing system of the present disclosure attempts to solve. 1 is a block diagram illustrating a configuration example of an image processing system that is an embodiment of the present disclosure. FIG. 6 is a diagram illustrating a method of holding reference plane candidates in an encoding unit. FIG. 3 is a diagram illustrating the operation of the encoding device when an error notification is received from the decoding device. It is a flowchart explaining encoding transmission processing performed by an encoding device. FIG. 3 is a diagram illustrating an example of reception error control when there are multiple destinations. FIG. 7 is a diagram illustrating an example of dynamic control of reference plane candidate intervals when a destination is added or deleted. FIG. 3 is a diagram illustrating an example where there are variations in transmission delay time and reliability of transmission paths among a plurality of destinations. FIG. 6 is a diagram illustrating an example of control when there are variations in transmission delay time and reliability of transmission paths among a plurality of destinations. FIG. 1 is a block diagram illustrating a configuration example of an embodiment of a computer to which the technology of the present disclosure is applied.

Hereinafter, embodiments for implementing the technology of the present disclosure (hereinafter referred to as embodiments) will be described with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals and redundant explanation will be omitted. The explanation will be given in the following order.
1. Problem 2 that the image processing system of the present disclosure attempts to solve. Block diagram 3 of the image processing system of the present disclosure. Example 4 of reference plane holding control according to transmission delay. Example 5 of controlling the reference plane candidate interval according to the transmission delay when an error occurs. Flowchart of encoded transmission processing 6. Example 7 of reception error control when there are multiple destinations. Example 8 of reference plane retention control when there is bias in the transmission delay situation at the destination. Summary 9. Computer configuration example

<1. Problems to be solved by the image processing system of the present disclosure>
First, the problem to be solved by the image processing system of the present disclosure will be described with reference to FIGS. 1 to 3.

The AVC (Advanced Video Coding) method, HEVC (High Efficiency Video Coding) method, etc. are widely used as compression encoding methods used to deliver video images in real time. In the AVC and HEVC compression encoding systems, there are three types of pictures: I pictures, P pictures, and B pictures, similar to the MPEG (Motion Picture Experts Group)-1/2/4 system. An I-picture is a picture obtained by independently encoding only that image, regardless of the preceding and succeeding images. A P picture is a picture obtained by forward predictive coding between images. A B picture is a picture obtained by predictive coding from both the past and the future.

For B pictures, the encoding order and display order are different, so it is necessary to rearrange the images. When delivering video images with low delay, encoding is performed without image rearrangement, and in this case, the distance between the image to be encoded and the reference picture (hereinafter referred to as picture distance) is close to each other. It is common to use an image of 1 as a reference picture.

FIG. 1 shows an example of a moving image in which the immediately previous image is encoded as a reference picture.

The moving images in FIG. 1 are sequentially generated as images fr1, fr2, fr3, fr4, . . . Each picture fr is encoded by a P picture using the immediately previous picture as a reference picture. When image fr1 is the first image of a moving image, image fr1 becomes an I picture.

In addition, in order to achieve low-delay delivery, it is necessary to reduce the buffer size of the encoding device and decoding device, and for this purpose, encoding is required to smooth the amount of code generated for each image. In some cases, the amount of generated code is smoothed at the slice level in order to further reduce the delay.

If an error occurs in the transmission path in such a low-latency streaming environment, it is necessary to restore normality, but as mentioned above, the amount of code generated is smoothed and the buffer for transmission and reception is minimized, so the amount of code is reduced. It is not possible to insert a large I-picture and restore the image quality while maintaining the image quality, so the encoding device must insert a small-sized I-picture that fits in the buffer.

An example of recovering from an error by inserting a small-sized I picture that fits in the buffer will be described with reference to FIGS. 2 and 3.

As shown in FIG. 2, a predetermined moving image is encoded in the encoding device 11, and the encoded bitstream is transmitted to three decoding devices 31-1 to 31-3 via the network 21. be done. Here, an error occurs in a predetermined image of the bitstream received by the decoding device 31-1, and the decoding device 31-1 requests a small-sized I picture that fits in the buffer to the encoding device 11, which is the transmission source. Suppose you send an insert request.

More specifically, as shown in FIG. 3, the moving image is sequentially encoded as images fr11, fr12, fr13, fr14, . . . using P pictures, and decoding devices 31-1 to 31-3 sent to. The decoding device 31-1 detects an error in the image fr13, and transmits a request to the encoding device 11, which is the transmission source, to insert an I picture of a small size that fits in the buffer.

At the timing when the encoding device 11 receives the I-picture insertion request, the encoding device 11 switches to encoding the next image fr using a small-sized I-picture that fits in the buffer. In the example of FIG. 3, the image fr17 is generated by a small-sized I picture and transmitted to the decoding devices 31-1 to 31-3. In the decoding device 31-1, images fr14 to fr16 from the image fr13 in which the error occurs until the image fr17 based on the I picture arrives are corrupted images because they refer to the error image. Furthermore, since the image fr17 is an I picture of a small size that fits in the buffer, the image quality has deteriorated. A period of several frames is required to recover from this deterioration in image quality. In the example of FIG. 3, the image quality is restored to that before the error occurred in image fr21, which is four frames after image fr17.

In a one-to-many distribution system as shown in FIG. 2, the deterioration in image quality caused by dealing with an error that occurred in the decoding device 31-1 is caused by the deterioration of image quality caused by dealing with the error that occurred in the decoding device 31-1. It also affects. That is, the decoding devices 31-2 and 31-3 also receive images fr17 to fr20 with degraded image quality.

As a method for recovering from an error to avoid image quality deterioration, it is conceivable to switch the reference picture to refer to an image that skips over the image in which the error occurred, instead of inserting an I picture. For example, instead of encoding the image fr17 in FIG. 3 as an I picture, it may be possible to encode it as a P picture by referring to the image fr12 before the image fr13 in which the error occurred.

However, in this case, it is necessary to always hold an image that jumps over the image in which the error has occurred as a reference picture candidate (hereinafter referred to as a reference plane candidate). In network distribution such as streaming, the delay time until reaching the decoding device 31 is not uniquely determined, so it is difficult to estimate the number of reference plane candidates, period, etc. to be retained for error recovery. If held redundantly, a large frame memory will be required in the encoding device 11 and decoding device 31. There is also a method of retaining reference plane candidates for a long period of time using a long-term reference picture function such as the AVC method or HEVC method, and updating them periodically. Since it will be referred to, it will be disadvantageous in terms of image quality.

The encoding device of the image processing system of the present disclosure described below encodes an image as a P picture that fits in a small buffer size, and when an error occurs on the receiving side, an image that jumps over the image in which the error occurred is encoded. Encode as a reference picture. The encoding device saves frame memory and transmits moving images with reduced image quality deterioration by controlling how reference plane candidates are held according to the delay status of the network that transmits the stream.

<2. Block diagram of the image processing system of the present disclosure>
FIG. 4 is a block diagram illustrating a configuration example of an image processing system according to an embodiment of the present disclosure.

The image processing system 50 in FIG. 4 includes an encoding device 51 and a decoding device 52. In the image processing system 50 , a bitstream that is moving image data encoded by the encoding device 51 is transmitted (distributed) to the decoding device 52 via the network 53 .

The encoding device 51 includes a reference structure control section 71, an encoding section 72, a storage section 73, a transmitting section 74, and a receiving section 75.

The reference structure control unit 71 monitors the transmission delay status of the moving image bitstream and dynamically controls the method of holding reference plane candidates in the encoding unit 72. Specifically, the reference structure control unit 71 acquires transmission delay information supplied from the decoding device 52 via the reception unit 75, generates reference plane control information based on the acquired transmission delay information, and performs encoding. 72. The reference plane control information includes the reference plane candidate interval and the number of reference plane candidates. The reference plane candidate interval and the number of reference plane candidates are information that controls the method of holding reference plane candidates in the encoding unit 72. Further, when a reception error occurs in the decoding device 52, the reference structure control unit 71 sends a reference plane candidate instruction for recovering from the error (hereinafter referred to as an error recovery reference plane candidate instruction) to the encoding unit 52. supply to.

The encoding unit 72 encodes the input moving image using a predetermined encoding method based on the reference plane control information supplied from the reference structure control unit 71, and supplies the resulting bit stream to the transmitting unit 74. do. Here, the encoding method (encoding method) performed by the encoding unit 72 is not particularly limited as long as there are two types of pictures, I pictures and P pictures, in which image rearrangement does not occur. , AVC (Advanced Video Coding) method, HEVC (High Efficiency Video Coding) method, etc. can be adopted.

The encoding unit 72 encodes images that are sequentially input as moving images without rearranging the images, on the premise of low-delay transmission. Specifically, the encoding unit 72 uses a plurality of images encoded before the image to be encoded (hereinafter referred to as the current picture) among the sequentially input images as reference plane candidates in the storage unit 73. Remember it. The reference plane candidate is a reference picture (reference plane) candidate, and the encoding unit 72 uses a predetermined one of the plurality of images stored as the reference plane candidate as a reference picture and encodes it as a P picture.

The encoding unit 72 determines a plurality of images to be stored in the storage unit 73 as reference plane candidates based on the reference plane candidate interval and the number of reference plane candidates supplied from the reference structure control unit 71. The reference plane candidate interval represents the interval between images stored in the storage unit 73 as reference plane candidates. The number of reference plane candidates represents the number of images to be stored in the storage unit 73 as reference plane candidates. The encoding unit 72 also updates the image of the reference plane candidate stored in the storage unit 73 as the image is encoded. In updating the reference plane candidates, the oldest image is discarded (deleted), and the newest image is stored in the vacant area.

The storage unit 73 is a frame memory that stores multiple images as reference plane candidates.

The transmitter 74 transmits the bitstream of the moving image supplied from the encoder 72 to the decoder 52 via the network 53.

The receiving unit 75 receives transmission delay information transmitted from the decoding device 52 via the network 53 and supplies it to the reference structure control unit 71. Further, the receiving unit 75 receives an error notification transmitted from the decoding device 52 when an error occurs in the decoding device 52, and supplies it to the reference structure control unit 71. The error notification includes, for example, the frame number of the image in which the error occurred and an error recovery request requesting recovery from the error.

The decoding device 52 includes a receiving section 91, a decoding section 92, a storage section 93, and a transmitting section 94.

The receiving unit 91 receives the bitstream of the moving image transmitted from the transmitting unit 74 of the encoding device 51 and supplies it to the decoding unit 92.

The decoding unit 92 decodes the bitstream of the moving image supplied from the receiving unit 91 and outputs the resulting moving image. Reference plane candidates required for decoding are stored in the storage unit 93 as appropriate. The decoder 92 detects the transmission delay status of the bitstream and supplies the detection result to the transmitter 94 as transmission delay information. Furthermore, if there is an error in the bitstream supplied from the receiving section 91, the decoding section 92 supplies an error notification to the transmitting section 94.

The storage unit 93 is a frame memory that stores a plurality of images as reference plane candidates. When the reference plane candidates are updated, the oldest image is discarded (deleted), and the newest image is stored in the space vacated by this.

The transmitting unit 94 transmits the transmission delay information and error notification supplied from the decoding unit 92 to the encoding device 51 via the network 53.

The network 53 is any communication network, and may be a wired communication network, a wireless communication network, or both. Further, the network 53 may be configured with one communication network, or may be configured with a plurality of communication networks. The network 53 is, for example, the Internet, a public telephone line, a wide area communication network for wireless mobile devices such as a so-called 4G line or 5G line, a WAN (Wide Area Network), a LAN (Local Area Network), or a Bluetooth (registered trademark) standard. compliant wireless communication networks, short-range wireless communication channels such as NFC (Near Field Communication), infrared communication channels, HDMI (registered trademark) (High-Definition Multimedia Interface) and USB (Universal Serial Bus) ), etc., or any other communication network or channel of any communication standard.

The image processing system 50 in FIG. 4 is configured as described above.

<3. Example of reference plane retention control according to transmission delay>
With reference to FIG. 5, a method of holding reference plane candidates in the encoding unit 72, which is dynamically controlled by the reference structure control unit 71 according to the bitstream transmission delay status, will be described.

The reference structure control unit 71 decreases the reference plane candidate interval when the transmission delay time is short, and increases the reference plane candidate interval when the transmission delay time is long.

A in FIG. 5 shows an example of reference plane candidate intervals when the transmission delay time is short. In A of FIG. 5, the reference plane candidate interval is set to "0", which is the minimum value.

B in FIG. 5 shows an example of reference plane candidate intervals when the transmission delay time is long. In B of FIG. 5, the reference plane candidate interval is set to "3".

Note that in both A and B of FIG. 5, the current picture is image P16, and the number of reference plane candidates is "4".

As shown in FIG. 5A, when the current picture is image P16 and the reference plane candidate interval is set to "0", the storage unit 73 stores four images P12 to P15 as reference plane candidates. images are stored. The interval between the images P12 to P15, which are stored as reference plane candidates in the storage unit 73, is "0".

The encoding unit 72 selects the image with the smallest temporal distance (picture distance) to the current picture, that is, the immediately preceding image P15, as a reference picture from among the four images P12 to P15 stored in the storage unit 73. , encodes the image P16, which is the current picture.

On the other hand, as shown in FIG. 5B, when the current picture is image P16 and the reference plane candidate interval is set to "3", the storage unit 73 stores images P3 and P7 as reference plane candidates. , P11, and P15 are stored. The interval between images P3, P7, P11, and P15, which are stored in the storage unit 73 as reference plane candidates, is "3".

Of the four images P3, P7, P11, and P15 stored in the storage unit 73, the encoding unit 72 selects an image having the closest temporal distance (picture distance) to the current picture, that is, the immediately previous image P15. is selected as a reference picture, and the current picture, image P16, is encoded.

The reason why the reference plane candidate interval is set large when the transmission delay time is long is that, as will be described later with reference to FIG. This is to enable encoding to be performed by selecting as a reference picture an image that is older in time than the image in which the error has occurred. When an error occurs, an image that is older in time than the image in which the error occurred must be stored in the storage unit 73. Therefore, the reference structure control unit 71 selects a reference surface candidate when the transmission delay time is long. Control is performed such that the interval is set large and older images are stored in the storage unit 73.

<4. Example of controlling the reference plane candidate interval according to the transmission delay when an error occurs>
Next, referring to FIG. 6, when a reception error occurs in the decoding device 52, in other words, when an error notification is received from the decoding device 52, the control of the reference structure control section 71 and the control of the encoding section 72 The encoding of is explained.

The example in FIG. 6 is an example in which the setting values for the reference plane candidate interval and the number of reference plane candidates are the same as in A of FIG. 5, that is, the reference plane candidate interval is "0" and the number of reference plane candidates is "4".

A in FIG. 6 shows reference plane candidates stored in the storage unit 73 and reference pictures selected from among them when the current picture is image P10.

When the current picture is image P10, images P6 to P9 are stored in the storage unit 73 as reference plane candidates. The encoding unit 72 selects the immediately preceding image P9 as a reference picture, and encodes the image P10, which is the current picture.

B in FIG. 6 shows the reference plane candidates stored in the storage unit 73 and the reference picture selected from them when the current picture is the image P11.

When the current picture is image P11, the reference plane candidates stored in the storage unit 73 have been updated to images P7 to P10. The encoding unit 72 selects the immediately preceding image P10 as a reference picture, and encodes the image P11, which is the current picture.

Here, assume that during encoding of image P11, reference structure control unit 71 receives an error notification from decoding device 52 indicating that an error has occurred in image P9. In this case, when encoding the next image P12, normally the immediately preceding image P11 is used as the reference picture, but since an error has occurred in image P9, images P9 to P11 are used as the reference picture. I can not use it.

Therefore, the reference structure control unit 71 supplies the encoding unit 72 with an instruction to exclude images P9 to P11 from the reference plane candidates as a reference plane candidate instruction for recovering from the error (error recovery reference plane candidate instruction). . The encoding unit 72 excludes the designated images P9 to P11 from among the images P8 to P11 stored in the storage unit 73 from the reference plane candidates, and determines which temporal distance is the current one among the remaining reference plane candidates. Select the image closest to the picture. In other words, the encoding unit 72 selects the image that skips over the image P9 in which the error has occurred and is closest to the current picture. As shown in C of FIG. 6, the reference plane candidates stored in the storage unit 73 have been updated to images P8 to P11, and among the remaining reference plane candidates, the image whose temporal distance is closest to the current picture is This is image P8. The encoding unit 72 selects image P8 as a reference picture and encodes image P12, which is the current picture.

The error recovery reference plane candidate instruction is applied only to the current picture at the timing when the instruction is supplied. Therefore, when the current picture becomes image P13, as shown in FIG. Image P13, which is a picture, is encoded.

As described above, when the reference structure control unit 71 receives an error notification indicating that an error has occurred in a predetermined image from the decoding device 52, the reference structure control unit 71 generates an error message that excludes images after the image in which the error has occurred from reference plane candidates. A return reference plane candidate instruction is supplied to the encoding unit 72.

In the above example, the reference structure control unit 71 supplies the reference plane candidates to be excluded from among the reference plane candidates stored in the storage unit 73 as error recovery reference plane candidate instructions. The reference plane candidate may be supplied as an error recovery reference plane candidate instruction. In the example of C in FIG. 6, instead of an instruction to exclude images P8 to P11 from reference plane candidates, an instruction to select image P8 may be used as an error recovery reference plane candidate instruction. Alternatively, the reference structure control unit 71 supplies the frame number of the image in which the error occurred to the encoding unit 72 as an error recovery reference plane candidate instruction, and selects which of the reference plane candidates stored in the storage unit 73 is the error recovery reference plane candidate. The encoding unit 72 may decide whether to use the picture as a reference picture for restoration. The reference plane candidate instruction may be an instruction for the encoding unit 72 to select a reference plane candidate before the image in which the error has occurred as a reference picture.

When the reference picture selection method is changed from the image immediately before the current picture, which is the normal selection method, to a predetermined number of images before the picture in which the error has occurred, the decoding device 52 changes the reference picture selection method. need to be informed. Changes in the reference picture selection method can be notified using, for example, an MMCO (Memory Management Control Operation) command in the AVC encoding method. Furthermore, in the HEVC encoding method, notification can be performed using RPS (Reference Picture Set).

According to the MMCO command, for the reference plane candidates stored in the storage unit 73,
・Setting a short-term reference picture as a non-reference picture,
- Setting a short-term reference picture as a long-term reference picture by assigning a long-term frame index, which is a reference index for managing long-term reference pictures, to the short-term reference picture;
・Setting the maximum value of long-term frame index,
- It is possible to set all reference pictures to non-reference pictures, etc.

Therefore, in the example of C in FIG. 6, images P9 to P11 can be excluded from the reference plane candidates by the MMCO command. Reference pictures are specified using a reference picture list.

In the HEVC method, short_term_ref_pic_set(), which is a function of RPS (Reference Picture Set), includes delta_idx_minus1, which is reference picture identification information that identifies a reference picture. Specifically, delta_idx_minus1 is the value obtained by subtracting 1 from the value obtained by subtracting the coding number of the reference picture from the coding number (Coding Order) of the image to be coded. In the example of C in FIG. 6, a value specifying image P8 is stored in delta_idx_minus1.

<5. Flowchart of encoded transmission process>
Next, the encoding and transmitting process performed by the encoding device 51 will be described with reference to the flowchart in FIG. This process corresponds to the process of encoding and transmitting a predetermined image as a current picture among a plurality of images forming an input moving image. For example, when a current picture is input to the encoding device 51, is started when. Note that the number of reference plane candidates is a fixed value (for example, "4") as in the above example.

First, in step S21, the reference structure control unit 71 determines whether an error notification indicating that an error has occurred has been received from the decoding device 52.

If it is determined in step S21 that an error notification has not been received from the decoding device 52, the process proceeds to step S23. On the other hand, if it is determined in step S21 that an error notification has been received from the decoding device 52, the process proceeds to step S22.

In step S22 when it is determined that an error notification has been received, the reference structure control unit 71 sends an instruction to exclude from the image in which the error has occurred to the image immediately before the current picture from the reference plane candidates. It is supplied to the encoding unit 72 as an instruction.

In step S23, the reference structure control unit 71 determines whether transmission delay information has been received from the decoding device 52.

If it is determined in step S23 that the transmission delay information has not been received from the decoding device 52, the process proceeds to step S25. On the other hand, if it is determined in step S23 that transmission delay information has been received from the decoding device 52, the process proceeds to step S24.

In step S24 when it is determined that transmission delay information has been received, the reference structure control unit 71 updates the reference plane candidate interval based on the transmission delay information and supplies it to the encoding unit 72. Note that if there is no change in the reference plane candidate interval, the instruction of the reference plane candidate interval to the encoding unit 72 can be omitted. That is, the reference plane candidate interval may be instructed to the encoding unit 72 only when there is a change in the reference plane candidate interval.

In step S25, the encoding unit 72 updates the reference plane candidates stored in the storage unit 73. The encoding unit 72 stores the image immediately before the current picture in the storage unit 73 as the latest reference plane candidate, and deletes the oldest reference plane candidate from the storage unit 73.

In step S26, the encoding unit 72 selects the reference plane candidate with the closest temporal distance to the current picture as the reference picture.

In step S27, the encoding unit 72 encodes the current picture using a predetermined encoding method such as the AVC method or the HEVC method. The bitstream obtained as a result of encoding is supplied to the transmitter 74.

In step S28, the transmitter 74 transmits the bitstream supplied from the encoder 72 to the decoder 52 via the network 53.

The encoding and transmitting process in FIG. 7 is executed as described above, and is repeatedly executed for each of the plurality of images that make up the moving image.

The encoding and transmitting process in FIG. 7 has been described as a series of processes for convenience of explanation, but the processes in steps S23 and S24 for changing the reference plane candidate interval based on transmission delay information are similar to those in steps S21, S22, Further, it can be executed independently of the processes of S25 to S28. The encoding unit 72 may store the reference plane candidates using the changed reference plane candidate retention method from the next update of the reference plane candidates in which the reference plane candidate intervals have been changed.

Furthermore, in the above example, the number of reference plane candidates is not changed, but the number of reference plane candidates may also be changed based on the transmission delay information. In this case, if the transmission delay time is short, the number of reference plane candidates is decreased, and if the transmission delay time is long, the number of reference plane candidates is increased.

Furthermore, in the example described above, an example was described in which encoding is performed using only one image as a reference picture, but a plurality of images are selected as reference pictures from among the reference plane candidates stored in the storage unit 73. It may also be encoded.

<6. Example of reception error control when there are multiple destinations>
In the example described above, the transmission destination to which the bitstream of the moving image generated by the encoding device 51 is transmitted is one decoding device 52. However, in the following, the case where there are multiple decoding devices 52 as the transmission destination is explained. An example of control for recovering from a reception error will be described.

FIG. 8 shows an example in which a bitstream of a moving image generated by the encoding device 51 is transmitted to three decoding devices 52-1 to 52-3.

The transmission paths through which the video bitstream is transmitted to each of the three destinations may have short delay times or long paths; therefore, the transmission delay times of each decoding device 52 are not necessarily the same. do not have. In the example of FIG. 8, the transmission delay time of the decoding devices 52-1 to 52-3 is shown as "Delay". It is assumed that the transmission delay times of the decoding devices 52-1 to 52-3 are Delay=2, Delay=1, and Delay=5, respectively. For simplicity, the unit of transmission delay time (Delay) is assumed to be a frame.

The encoding device 51 holds reference plane candidates so that even if a reception error occurs in any decoding device 52, it can recover from the error. In the example of FIG. 8, the reference structure control unit 71 of the encoding device 51 controls the reference plane candidate holding method to match Delay=5, which has the longest transmission delay time.

As mentioned above, using a temporally older image as a reference picture for error recovery means referring to an image that is temporally distant, which is disadvantageous in terms of image quality. Since the bitstream generated for error recovery is transmitted to all destinations, it is better to reduce the number of frames to be rewinded to a temporally older image as a reference picture.

For example, if an error occurs in the transmission path to the decoding device 52-1 and an error notification is sent from the decoding device 52-1, the reference structure control unit 71 of the encoding device 51 An error recovery reference plane candidate instruction is supplied to the encoding unit 72 so that the reference plane candidate is used as a reference picture. For example, if an error occurs in the transmission path to the decoding device 52-3 and an error notification is sent from the decoding device 52-3, the reference structure control unit 71 of the encoding device 51 An error recovery reference plane candidate instruction is supplied to the encoding unit 72 so as to use the reference plane candidate as the reference picture.

As described above, the reference structure control unit 71 of the encoding device 51 acquires transmission delay information from each of the plurality of decoding devices 52 that are transmission destinations, and selects reference plane candidates according to the transmission delay information with the longest transmission delay time. Determine how to retain the Then, by controlling the reference picture to be switched according to the transmission delay time of the decoding device 52 where the error occurred, it is possible to recover from the error while suppressing deterioration in image quality.

During distribution, if a predetermined decoding device 52 is deleted from the destination, or conversely, a new destination is added, the maximum value of the transmission delay time for the entire distribution target will also change. The conversion device 51 also changes the method of holding reference plane candidates.

FIG. 9 shows an example of dynamic control of the reference plane candidate interval when a destination is added or deleted. In FIG. 9, a rectangular frame is attached to the maximum transmission delay time among the transmission delay times (Delay) of the plurality of decoding devices 52 to be transmitted. In the example of FIG. 9, the number of reference plane candidates is not changed and is set to a fixed value (for example, "4").

In the state on the left side of the upper row (above the broken line) of FIG. 9, there are three decoding devices 52-1 to 52-3 as transmission destinations. The transmission delay times of the decoding devices 52-1 to 52-3 are Delay=1, Delay=8, and Delay=4, respectively. The maximum transmission delay time is Delay=8 of the decoding device 52-2, and the reference structure control unit 71 of the encoding device 51 adjusts the reference plane candidate interval according to Delay=8, which is the maximum transmission delay time. It is set to "5".

After that, if the decoding device 52-2 is deleted from the destinations, there will be two destinations, the decoding devices 52-1 and 52-3, as shown in the upper right side of FIG. 9, and the maximum transmission delay time will be , the Delay of the decoding device 52-3 is 4. The reference structure control unit 71 of the encoding device 51 changes the reference plane candidate interval to "3" in accordance with Delay=4, which is the maximum transmission delay time.

The state on the left side of the lower row of FIG. 9 (below the broken line) shows the same state as the right side of the upper row of FIG. 9. That is, the transmission destinations are two decoding devices 52-1 and 52-3, the maximum transmission delay time is Delay=4 for the decoding device 52-3, and the reference plane candidate interval is set to "3". ing.

Thereafter, as shown on the right side of the bottom row of FIG. 9, a new decoding device 52-4 has joined as a transmission destination. There are three destinations: decoding devices 52-1, 52-3, and 52-4, and the maximum transmission delay time is Delay=6 for decoding device 52-4. In this case, the reference structure control unit 71 of the encoding device 51 changes the reference plane candidate interval to "4" in accordance with Delay=6, which is the maximum transmission delay time.

As described above, the reference structure control unit 71 of the encoding device 51 determines whether a change in the destination, including at least one of an increase or a decrease in the number of destinations, has occurred, and delays transmission from each current destination. The transmission delay time is determined by acquiring the information. The reference structure control unit 71 updates the reference plane candidate interval when it is determined that a change in the transmission destination has occurred or when the transmission delay time has been changed.

In the example described above, the number of reference plane candidates is not changed and is set to a fixed value, and only the interval between reference plane candidates is changed according to the transmission delay situation at the destination. However, the number of reference plane candidates may also be changed depending on the transmission delay situation at the destination. For example, if the maximum value of the reference plane candidate interval is determined, and the transmission delay time becomes even longer with the reference plane candidate interval set to the maximum value, the reference structure control unit 71 Control to change the number. For example, if the maximum value of the reference plane candidate interval is "4" and the decoding devices 52-1, 52-3, and 52-4 shown on the right side of the lower part of FIG. When the time is changed from Delay=6 to Delay=8, the reference structure control unit 71 maintains the reference plane candidate interval as “4” and increases the number of reference plane candidates from the current “4” to “5”. Change to If the reference plane candidate interval becomes large, an image with a long temporal distance will be used as a reference picture at the time of error recovery. In this way, both the reference plane candidate interval and the number of reference plane candidates are controlled, and if the reference plane candidate interval has reached a predetermined value, the number of reference plane candidates can be increased to prevent transmissions in which an error has occurred. Depending on the destination, an image with a short temporal distance can be selected as a reference picture.

The reference structure control unit 71 sets the number of reference plane candidates as a fixed value in stages according to the transmission delay status of the destination, and sets the reference plane candidates as a fixed value in accordance with the transmission delay status of the destination at the fixed number of reference plane candidates. The spacing may be controlled. For example, if the transmission delay time of each destination is within the range of "1 to 5", the number of reference plane candidates is set to "4", and if the transmission delay time of each destination is within the range of "6 to 10", the number of reference plane candidates is set to "4". In this case, the number of reference plane candidates may be set to "5", and with the set number of reference plane candidates, the interval between reference plane candidates may be further controlled according to the transmission delay situation.

<7. Example of reference plane retention control when the transmission delay status of the destination is uneven>
By the way, when there are a plurality of decoding devices 52 as transmission destinations, the delay status of each transmission destination may be biased. Furthermore, there may be cases where the reliability of the transmission path varies.

FIG. 10 shows an example where there are variations in transmission delay time and reliability of transmission paths among multiple destinations.

In the example of FIG. 10, there are N decoding devices 52-1 to 52-N (N>1) as transmission destinations. The encoding device 51 transmits the generated bitstream of the moving image to the decoding devices 52-1 to 52-N via the network 53. The transmission delay time for the decoding device 52-1 is Delay=1, and the error rate representing the reliability of the transmission path to the decoding device 52-1 is _E1 . The transmission delay time for the decoding device 52-2 is Delay=2, and the error rate representing the reliability of the transmission path to the decoding device 52-2 is _E2 . The transmission delay time for the decoding device 52-N is Delay=10, and the error rate representing the reliability of the transmission path to the decoding device 52-N is E _n .

Regarding the transmission delay time and the reliability of the transmission path, it is assumed that the N decoding devices 52-1 to 52-N are divided into two groups: a group with short transmission delay time and a group with long transmission delay time. For example, the first group is composed of decoding devices 52-1 to 52-K (0<K<N), and is a group with a short transmission delay time of Delay=1 or 2. The second group is composed of decoding devices 52-(K+1) to 52-N, and is a group with a long transmission delay time of Delay=9 or 10. Regarding the reliability of the transmission path, if the first group with a short transmission delay time has a low error rate and the second group with a long transmission delay time has a high error rate, then on the other hand, if the first group with a short transmission delay time has a high error rate, There may be cases where the error rate of the first group is high, and the second group with a long transmission delay time has a low error rate.

In this way, when there are variations in the transmission delay time of multiple destinations and the reliability of the transmission path, if the value of the reference plane candidate interval is determined by focusing only on the maximum value of the transmission delay time, the delay time There is a concern that the quality of the recovery image for errors that occur on a short path may deteriorate. In other words, it is originally desirable to use images that are close in time as reference pictures, but because the value of the reference plane candidate interval is determined according to the maximum transmission delay time, images that are far away in time are not used as reference pictures. A situation may occur in which the picture must be used as a reference picture, and as a result, the quality of the image upon error recovery may deteriorate.

In response to such concerns, the encoding device 51 can prevent deterioration of image quality upon error recovery by controlling the method of holding reference plane candidates as follows.

Specifically, the reference structure control unit 71 of the encoding device 51 sets the reference plane candidate interval and the number of reference plane candidates respectively for a first group with a short transmission delay time and a second group with a long transmission delay time. control to hold two types of values corresponding to . Hereinafter, the reference plane candidate interval and the number of reference plane candidates of the first group will be referred to as the first reference plane candidate interval and the first reference plane candidate number, and the reference plane candidate interval and the number of reference plane candidates of the second group will be referred to as: These will be referred to as the second reference plane candidate interval and the number of second reference plane candidates.

In A of Figure 11, there are many destinations in the first group with short transmission delay times, and their transmission paths have high error rates, while there are few destinations in the second group, which has long transmission delay times, and their transmission paths have high error rates. An example of control when the error rate of the transmission path is low is shown.

For example, the reference structure control unit 71 sets the first reference plane candidate interval to "0" and the number of first reference plane candidates to "3" for the first group with a short transmission delay time, and sets the first reference plane candidate interval to "3" for the first group with a short transmission delay time. Reference plane control information in which the second reference plane candidate interval of the group is “5” and the number of second reference plane candidates is “2” is supplied to the encoding unit 72.

When the current picture is image P16, the storage unit 73 stores images P1, P7, and P13 to P15 as reference plane candidates. Three images P13 to P15 are reference plane candidates for the first group with short transmission delay time, and two images P1 and P7 are reference plane candidates for the second group with long transmission delay time. It is a candidate. In images P13 to P15, the reference plane candidate interval corresponds to the first reference plane candidate interval “0” and the number of reference plane candidates corresponds to the first reference plane candidate number “3”. In images P1 and P7, the reference plane candidate interval corresponds to the second reference plane candidate interval “5” and the number of reference plane candidates corresponds to the number of second reference plane candidates “2”.

B in Figure 11 shows that there are many destinations in the first group with short transmission delay times, and the error rate of their transmission paths is low, while there are few destinations in the second group, which has long transmission delay times, and their transmission paths have low error rates. An example of control when the error rate of the transmission path is high is shown.

For example, the reference structure control unit 71 sets the first reference plane candidate interval to "1" and the number of first reference plane candidates to "2" for the first group with a short transmission delay time, and sets the first reference plane candidate interval to "2" for the first group with a short transmission delay time, and sets the first reference plane candidate interval to "2" for the first group with a short transmission delay time. Reference plane control information that sets the second reference plane candidate interval of the group to “3” and the number of second reference plane candidates to “3” is supplied to the encoding unit 72.

When the current picture is image P16, the storage unit 73 stores images P1, P5, P9, P13, and P15 as reference plane candidates. The two images P13 and P15 are reference plane candidates for the first group with short transmission delay time, and the three images P1, P5, and P9 are reference plane candidates for the second group with long transmission delay time. is a reference plane candidate. In images P13 and P15, the reference plane candidate interval corresponds to the first reference plane candidate interval “1” and the number of reference plane candidates corresponds to the first reference plane candidate number “2”. In images P1, P5, and P9, the reference plane candidate interval corresponds to the second reference plane candidate interval “3” and the number of reference plane candidates corresponds to the second reference plane candidate number “3”.

As described above, the reference structure control unit 71 selects a group with a high error rate (low reliability) out of the first group, which is a low-delay transmission destination, and the second group, which is a high-delay transmission destination. The numbers of first and second reference plane candidates are controlled so that the number of reference plane candidates is large and the number of reference plane candidates of a group with a low error rate (high reliability) is small. The first and second reference plane candidate intervals are set according to the transmission delay time of each group.

As a result, if an error occurs in a group with a short transmission delay time, an image with a short temporal distance can be used as a reference picture, and even if an error occurs in a group with a long transmission delay time, an error occurs. able to cope with the return.

Note that in the above example, multiple destinations are divided into two groups and two types of reference plane candidate intervals and the number of reference plane candidates are determined. The destinations may be divided into three or more groups, and the reference plane candidate intervals and the number of reference plane candidates may be determined in accordance with the number of groups.

<8. Summary>
As described above, in the encoding device 51, the reference structure control unit 71 monitors the transmission delay status of the bit stream of the moving image of the decoding device 52, which is the transmission destination, and controls the encoding unit 72 according to the transmission delay status. By dynamically controlling the method of holding reference plane candidates, it is possible to recover from errors with minimal deterioration in image quality. It is possible to prevent reference plane candidates from being held redundantly for error recovery, and it is possible to reduce the capacity of the storage unit 73, which is a frame memory that stores reference plane candidates. If the transmission delay time changes, by changing the reference plane candidate interval and the number of reference plane candidates accordingly, you can save frame memory and select an appropriate reference plane candidate, allowing you to recover from errors. can be done.

Even when there is not just one destination but multiple destinations, changes in the destination, including at least one of an increase or decrease in the number of destinations, are monitored, and a method for holding reference plane candidates is determined in accordance with the longest transmission delay time. This makes it possible to select the optimal reference plane candidate and recover from an error using a small amount of frame memory. If there is a bias in the transmission delay status of destinations, by having multiple reference plane candidate intervals and the number of reference plane candidates, it is possible to select the appropriate one for the destination with short transmission delay time and the destination with long transmission delay time. Reference plane candidates can be selected and errors can be recovered.

The decoding device 52 transmits the transmission delay status of the bitstream of the received moving image to the encoding device 51, so that even if an error occurs in the received bitstream, the decoding device 52 can determine the transmission delay status appropriate for its own transmission delay status even if an error occurs in the received bitstream. A reference picture may be selected and an encoded bitstream may be received. In other words, it is possible to recover from the error with minimal deterioration in image quality. Furthermore, since an image with a short temporal distance is used as a reference picture in accordance with the transmission delay time, the capacity of the storage unit 93, which is a frame memory that stores reference plane candidates, can be reduced.

Therefore, the image processing system 50 in FIG. 4 can save frame memory and suppress image quality deterioration by reducing the number of retained reference plane candidates in low-delay bitstream distribution.

<9. Computer configuration example>
The series of processes described above can be executed by hardware or software. When a series of processes is executed by software, the programs that make up the software are installed on the computer. Here, the computer includes a microcomputer built into dedicated hardware, and a general-purpose personal computer that can execute various functions by installing various programs.

FIG. 12 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program.

In a computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are interconnected by a bus 104.

An input/output interface 105 is further connected to the bus 104. An input section 106 , an output section 107 , a storage section 108 , a communication section 109 , and a drive 110 are connected to the input/output interface 105 .

The input unit 106 includes a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like. The output unit 107 includes a display, a speaker, an output terminal, and the like. The storage unit 108 includes a hard disk, a RAM disk, a nonvolatile memory, and the like. The communication unit 109 includes a network interface and the like. The drive 110 drives a removable medium 111 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 101 executes the above-described series by, for example, loading a program stored in the storage unit 108 into the RAM 103 via the input/output interface 105 and the bus 104 and executing it. processing is performed. The RAM 103 also appropriately stores data necessary for the CPU 101 to execute various processes.

A program executed by the computer (CPU 101) can be provided by being recorded on a removable medium 111 such as a package medium, for example. Additionally, programs may be provided via wired or wireless transmission media, such as local area networks, the Internet, and digital satellite broadcasts.

In the computer, the program can be installed in the storage unit 108 via the input/output interface 105 by installing the removable medium 111 into the drive 110. Further, the program can be received by the communication unit 109 via a wired or wireless transmission medium and installed in the storage unit 108. Other programs can be installed in the ROM 102 or the storage unit 108 in advance.

Note that the program executed by the computer may be a program in which processing is performed chronologically in accordance with the order described in this specification, in parallel, or at necessary timing such as when a call is made. It may also be a program that performs processing.

In this specification, steps described in a flowchart may be performed chronologically in the order described, or may not necessarily be performed chronologically, but may be performed in parallel or when called. It may be executed at any necessary timing.

In this specification, a system means a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are in the same casing. Therefore, multiple devices housed in separate casings and connected via a network, and a single device with multiple modules housed in one casing are both systems. .

The embodiments of the present disclosure are not limited to the embodiments described above, and various changes can be made without departing from the gist of the technology of the present disclosure.

For example, a combination of all or part of the embodiments described above can be adopted.

The technology of the present disclosure can take a cloud computing configuration in which one function is shared and jointly processed by multiple devices via a network.

Each step explained in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

Note that the effects described in this specification are merely examples and are not limited, and there may be effects other than those described in this specification.

Note that the technology of the present disclosure can take the following configuration.
(1)
A plurality of images encoded before a predetermined image of a moving image to be encoded are stored in the storage unit as reference plane candidates based on reference plane control information supplied from the outside, and the plurality of images are the reference plane candidates. An encoding device comprising: an encoding unit that encodes the predetermined image using any one of the plurality of images as a reference picture.
(2)
The encoding unit acquires a reference plane candidate interval representing an interval between images to be stored in the storage unit from the outside as the reference plane control information, and determines the reference plane candidate based on the reference plane candidate interval. The encoding device according to (1) above.
(3)
The encoding unit acquires the number of reference plane candidates representing the number of images to be stored in the storage unit from the outside as the reference plane control information, and acquires the number of reference plane candidates from the outside based on the reference plane candidate interval and the number of reference plane candidates. The encoding device according to (2) above, which determines a reference plane candidate.
(4)
The encoding unit selects, as a reference picture, an image whose temporal distance is closest to the predetermined image from among the plurality of images stored in the storage unit as the reference plane candidate, and encodes the image. The encoding device according to any one of (1) to (3) above.
(5)
When a reference plane candidate instruction indicating a predetermined reference plane candidate is supplied from the outside, the encoding unit is configured to select one of the plurality of images stored in the storage unit based on the reference plane candidate instruction. The encoding device according to any one of (1) to (4), wherein one of the pictures is selected as a reference picture and encoded.
(6)
The encoding device according to (5), wherein the reference plane candidate instruction includes an instruction to exclude a predetermined image among the plurality of images stored in the storage unit from the reference plane candidates.
(7)
The encoding device according to (5), wherein the reference plane candidate instruction includes an instruction to specify an image that can be selected as a reference picture from among the plurality of images stored in the storage unit.
(8)
The encoding unit encodes the predetermined image using an AVC method,
The encoding device according to (5), wherein the reference picture selection method is notified to the destination decoding device using an MMCO command.
(9)
The encoding unit encodes the predetermined image using the HEVC method,
The encoding device according to (5) above, wherein the reference picture selection method is notified to the destination decoding device using RPS.
(10)
The encoding device is
A plurality of images encoded before a predetermined image of a moving image to be encoded are stored in the storage unit as reference plane candidates based on reference plane control information supplied from the outside, and the plurality of images are the reference plane candidates. An encoding method in which the predetermined image is encoded using any one of the plurality of images as a reference picture.
(11)
Based on transmission delay information from a destination that transmits a bitstream containing an encoded moving image, which of a plurality of images encoded before a predetermined image of the moving image to be encoded is selected as a reference plane candidate. An encoding device comprising: a control unit that determines reference plane control information for controlling whether to store it in a storage unit and supplies it to an encoding unit that encodes the predetermined image.
(12)
The encoding device according to (11), wherein the control unit supplies a reference plane candidate interval representing an interval between images to be stored in the storage unit to the encoding unit as the reference plane control information.
(13)
The encoding device according to (12), wherein the control unit supplies, as the reference plane control information, the number of reference plane candidates representing the number of images to be stored in the storage unit to the encoding unit.
(14)
The control unit decreases the reference plane candidate interval when the transmission delay time as the transmission delay information is short, and increases the reference plane candidate interval when the transmission delay time is long. (12) or (13) above. The encoding device described in .
(15)
The control unit changes the number of reference plane candidates, which is the number of images to be stored in the storage unit, when the transmission delay time becomes larger with the reference plane candidate interval set to the maximum value. The encoding device according to any one of (12) to (14).
(16)
When there are multiple destinations, the control unit acquires the transmission delay information from each of the multiple destinations, and determines the reference plane control information according to the transmission delay information with the longest delay time. (11) above. The encoding device according to any one of (15) to (15).
(17)
The control unit determines whether a change in the destination, including at least one of an increase or a decrease in the destination, has occurred, and updates the reference plane control information based on the determination that the change in the destination has occurred. The encoding device according to any one of (11) to (16) above.
(18)
The control unit determines at least two types of the reference plane control information, one for a low-delay transmission destination and one for a high-delay transmission destination, and supplies it to the encoding unit. The encoding device described.
(19)
When the control unit receives an error notification indicating that an error has occurred from the destination, the control unit issues a reference plane candidate instruction to select an image of the reference plane candidate that is earlier than the image in which the error has occurred as a reference picture. , the encoding device according to any one of (11) to (18).
(20)
The encoding device is
Based on transmission delay information from a destination that transmits a bitstream containing an encoded moving image, which of a plurality of images encoded before a predetermined image of the moving image to be encoded is selected as a reference plane candidate. An encoding method, comprising: determining reference plane control information for controlling whether to store it in a storage unit, and supplying the information to an encoding unit that encodes the predetermined image.

50 Image processing system, 51 Encoding device, 52 Decoding device, 53 Network, 71 Reference structure control section, 72 Encoding section, 73 Storage section, 74 Transmission section, 75 Receiving section, 91 Receiving section, 92 Decoding section No. 93 Memory section, 94 transmitting section, 101 CPU, 102 ROM, 103 RAM, 106 input section, 107 output section, 108 storage section, 109 communication section, 110 drive

Claims

A plurality of images encoded before a predetermined image of a moving image to be encoded are stored in the storage unit as reference plane candidates based on reference plane control information supplied from the outside, and the plurality of images are the reference plane candidates. An encoding device comprising: an encoding unit that encodes the predetermined image using any one of the plurality of images as a reference picture.
The encoding unit acquires a reference plane candidate interval representing an interval between images to be stored in the storage unit from the outside as the reference plane control information, and determines the reference plane candidate based on the reference plane candidate interval. The encoding device according to claim 1.
The encoding unit acquires the number of reference plane candidates representing the number of images to be stored in the storage unit from the outside as the reference plane control information, and acquires the number of reference plane candidates from the outside based on the reference plane candidate interval and the number of reference plane candidates. The encoding device according to claim 2, wherein a reference plane candidate is determined.
The encoding unit selects, as a reference picture, an image whose temporal distance is closest to the predetermined image from among the plurality of images stored in the storage unit as the reference plane candidate, and encodes the image. The encoding device according to claim 1.
When a reference plane candidate instruction indicating a predetermined reference plane candidate is supplied from the outside, the encoding unit is configured to select one of the plurality of images stored in the storage unit based on the reference plane candidate instruction. The encoding device according to claim 1, wherein one of the pictures is selected as a reference picture and encoded.
The encoding device according to claim 5, wherein the reference plane candidate instruction includes an instruction to exclude a predetermined image among the plurality of images stored in the storage unit from the reference plane candidates.
The encoding device according to claim 5, wherein the reference plane candidate instruction includes an instruction to specify an image that can be selected as a reference picture from among the plurality of images stored in the storage unit.
The encoding unit encodes the predetermined image using an AVC method,
The encoding device according to claim 5, wherein the reference picture selection method is notified to the destination decoding device using an MMCO command.
The encoding unit encodes the predetermined image using the HEVC method,
The encoding device according to claim 5, wherein the reference picture selection method is notified to the destination decoding device using RPS.
The encoding device is
A plurality of images encoded before a predetermined image of a moving image to be encoded are stored in the storage unit as reference plane candidates based on reference plane control information supplied from the outside, and the plurality of images are the reference plane candidates. An encoding method in which the predetermined image is encoded using any one of the plurality of images as a reference picture.
Based on the transmission delay information from the transmission destination of the bitstream encoded with the moving image, the storage unit selects which of the plurality of images encoded before a predetermined image of the moving image to be encoded as a reference plane candidate. An encoding device, comprising: a control unit that determines reference plane control information for controlling whether to store the predetermined image in the image and supplies it to an encoding unit that encodes the predetermined image.
The encoding device according to claim 11, wherein the control unit supplies, as the reference plane control information, a reference plane candidate interval representing an interval between images to be stored in the storage unit to the encoding unit.
The encoding device according to claim 12, wherein the control unit supplies, as the reference plane control information, the number of reference plane candidates representing the number of images to be stored in the storage unit to the encoding unit.
The encoding according to claim 12, wherein the control unit decreases the reference plane candidate interval when the transmission delay time as the transmission delay information is short, and increases the reference plane candidate interval when the transmission delay time is long. Device.
The control unit changes the number of reference plane candidates, which is the number of images to be stored in the storage unit, when the transmission delay time becomes larger with the reference plane candidate interval set to a maximum value. Encoding device according to item 12.
When there are multiple destinations, the control unit acquires the transmission delay information from each of the multiple destinations, and determines the reference plane control information according to the transmission delay information with the longest delay time. The encoding device described.
The control unit determines whether a change in the destination, including at least one of an increase or a decrease in the destination, has occurred, and updates the reference plane control information based on the determination that the change in the destination has occurred. The encoding device according to claim 11.
The encoding device according to claim 11, wherein the control unit determines at least two types of the reference plane control information, one for a low-delay transmission destination and one for a high-delay transmission destination, and supplies the determined reference plane control information to the encoding unit.
When the control unit receives an error notification indicating that an error has occurred from the destination, the control unit issues a reference plane candidate instruction to select an image of the reference plane candidate that is earlier than the image in which the error has occurred as a reference picture. , the encoding device according to claim 11.
The encoding device is
Based on the transmission delay information from the transmission destination of the bitstream encoded with the moving image, the storage unit selects which of the plurality of images encoded before a predetermined image of the moving image to be encoded as a reference plane candidate. An encoding method in which reference plane control information for controlling whether to store the predetermined image is determined and supplied to an encoding unit that encodes the predetermined image.