EA045133B1 - METHOD, DEVICE AND MACHINE READABLE MEDIA CONTAINING A COMPUTER PROGRAM FOR FORMING A CONFIGURATION MESSAGE TO ENABLE THE DECODER TO RESTORE THE FIRST IMAGE REPRESENTATION WITH THE FIRST LEVEL OF QUALITY USING U CONFIGURATION DATA - Google Patents

Description

Field of technology

The invention relates to methods, devices, computer programs and computer-readable media. In particular, but not exclusively, the present invention relates to methods, apparatus, computer programs and computer-readable media for use in processing configuration information that relates to residual data useful for reconstructing an image representation with a relatively high level of quality.

State of the art

Signal compression and decompression is an important element in many known systems. Many types of signals, such as video, can be compressed and encoded for transmission, for example, over a data network. When such a signal is decoded, it may be desirable to improve the quality level of such a signal and/or to recover, to the greatest extent possible, as much information contained in the original signal as possible. Some known systems use scalable encoding methods. Scalable encoding involves encoding a signal along with information that allows the signal to be reconstructed at one or more different quality levels, for example, depending on the capabilities of the decoder and the available bandwidth. There are several limitations regarding signal reconstruction in a scalable encoding system. One such limitation is the amount of information that is stored, accessed and/or transmitted. The amount of information may vary, for example, depending on the desired level of quality of the reconstructed signal, the nature of the information that is used in the reconstruction, and/or how such information is configured. Another limitation is the decoder's ability to reliably process the information received. One factor that can affect the reliability of a decoder is its ability to handle errors and/or process received information that is unexpected, modified, and/or unrecognized by the decoder.

The essence of the invention

Various aspects of the present invention are set forth in the accompanying claims.

Additional features and advantages will become apparent from the following description of preferred embodiments, given by way of example only, with reference to the accompanying drawings.

Brief description of graphic materials

In fig. 1 is a block diagram of an example of a signal processing system in accordance with an embodiment of the present invention;

In fig. 2A and 2B show a block diagram of another example of a signal processing system in accordance with an embodiment of the present invention;

In fig. 3 is a diagram showing an example of a configuration message in accordance with an embodiment of the present invention;

In fig. 4 is a flowchart showing an example of a method according to an embodiment of the present invention;

In fig. 5 is a diagram showing another example of a configuration message in accordance with an embodiment of the present invention;

In fig. 6 is a flowchart showing another example of a method according to an embodiment of the present invention; And

In fig. 7 is a general block diagram of an example apparatus in accordance with an embodiment of the present invention.

Detailed description

In fig. 1 shows an example of a signal processing system 100. The signal processing system 100 is used for signal processing. Examples of signal types include, but are not limited to, video signals, image signals, audio signals, spatial signals such as those used in medical, scientific or holographic imaging, or other multidimensional signals.

The signal processing system 100 includes a first device 102 and a second device 104. The first device 102 and the second device 104 may have a client-server relationship, with the first device 102 functioning as a server device and the second device 104 functioning as a client device. The signal processing system 100 may include at least one additional device (not shown). The first device 102 and/or the second device 104 may include one or more components. One or more components may be implemented in hardware and/or software. In the signal processing system 100, one or more components may be located together or may be located remotely from each other. Examples of device types include, but are not limited to, computerized devices, handheld or laptop computers, tablets, mobile devices, gaming consoles, smart TVs, set-top boxes, augmented and/or virtual reality headsets, etc.

The first device 102 is communicatively connected to the second device 104 via a data network 106 . Examples of the data network 106 include, but are not limited to, the Internet, a local area network (LAN), and a wide area network (WAN). The first and/or second device 102, 104 may have a wired and/or wireless connection to the data network 106.

The first device 102 includes an encoder 108. The encoder 108 is configured to encode data contained in a signal, which is hereinafter referred to as signal data. For example, if such a signal is a video signal, the encoder 108 is configured to encode the video data. Video data contains a sequence of multiple images or frames. In addition to encoding signal data, encoder 108 may perform one or more additional functions. Encoder 108 may be implemented in various ways. For example, encoder 108 may be implemented in hardware and/or software.

Although in this example the first device 102 includes an encoder 108, in other examples the first device 102 is separate from the encoder 108. In such examples, the first device 102 is communicatively coupled to the encoder 108. The first device 102 may be implemented as one or more software functions and /or hardware modules.

The second device 104 includes a decoder 110. The decoder 110 is configured to decode signal data. In addition to decoding signal data, decoder 110 may perform one or more additional functions. Decoder 110 may be implemented in various ways. For example, decoder 110 may be implemented in hardware and/or software.

Although in this example the second device 104 includes a decoder 110, in other examples the second device 104 is separate from the decoder 110. In such examples, the second device 104 is communicatively coupled to the decoder 110. The second device 104 may be implemented as one or more software functions and/or or hardware modules.

The encoder 108 encodes the signal data and transmits the encoded signal data to the decoder 110 via the data network 106 . The decoder 110 decodes the received encoded signal data and generates the decoded signal data. The decoder 110 may output decoded signal data or data obtained using the decoded signal data. For example, decoder 110 may output such data for display on one or more display devices associated with second device 104.

In some of the examples described herein, encoder 108 provides to decoder 110 a representation of the signal at a given quality level and information that decoder 110 can use to reconstruct a representation of the signal at one or more higher quality levels. This information may be called recovery data. In some examples, view recovery involves obtaining a view that is not an exact copy of the original view. The extent to which such a representation matches the original may depend on various factors, including, but not limited to, quantization levels. A representation of a signal at a given level of quality can be thought of as a representation, version, or depiction of the data contained in the signal at a given level of quality. In some examples, the reconstruction data is included in the signal data that is encoded by the encoder 108 and transmitted to the decoder 110. For example, the reconstruction data may be in the form of metadata. In some examples, the reconstruction data is encoded and transmitted separately from the signal data.

The information that decoder 110 uses to reconstruct the signal's representation at one or more enhanced quality levels may include residual data, as described in more detail below. Residual data is an example of recovery data. The information that decoder 110 uses to restore a representation of the signal at one or more enhanced quality levels may also include configuration data related to processing of the residual data. The configuration data may indicate how the residual data was processed by the encoder 108 and/or how the residual data should be processed by the decoder 110. The configuration data may be provided to the decoder 110, for example, in the form of metadata.

In fig. 2A and 2B schematically illustrate an example of a signal processing system 200. The signal processing system 200 includes a first device 202 including an encoder and a second device 204 including a decoder. In each such first device 202 and second device 204, elements are shown at two logical levels. The two levels are separated by a dotted line. Elements of the first, highest level, are associated with data of a relatively high level of quality. Elements of the second, lowest level, are associated with data of a relatively low level of quality. Relatively high and relatively low quality levels refer to a multi-level hierarchy that has several levels of quality. In some examples, the multi-level hierarchy contains more than two quality levels. In each example, the first device 202 and the second device 204 may include more than two different layers. There may be one or more other levels above and/or below those depicted in FIG. 2A and 2B.

As shown in FIG. 2A, the first device 202 obtains a first image representation at a relatively high level 206 of quality. The representation of a given image is a representation of the data contained in the image. This image may represent a given frame

- 2 045133 videos. The first image representation at a relatively high quality level 206 will hereinafter be referred to as input data because, in this example, it is data provided as encoder input to the first device 202. The first device 202 may receive the input data 206. For example, the first device 202 may receive input data 206 from at least one other device. The first device 202 may be configured to receive successive pieces of input data 206, such as successive frames of video, and perform the operations described herein for each subsequent frame. For example, the video may contain frames F1, _F2 , ... F _T , and the first device 202 may process each of them in turn.

The first device 202 receives data 212 based on input data 206. In this example, data 212 based on input data 206 is a relatively low quality image representation 212. In this example, data 212 is obtained by performing a downsampling operation on input data 206 and is therefore referred to as downsampled data hereinafter. In other examples, data 212 is obtained by performing an operation other than downsampling on input data 206.

In this example, the downsampled data 212 is processed to generate processed data 213 with a relatively low level of quality. In other examples, the downsampled data 212 is not processed at a relatively low level of quality. Thus, the first device 202 may generate relatively low quality data, wherein such relatively low quality data comprises subsampled data 212 or processed data 213.

In some examples, generating the processed data 213 involves encoding the downsampled data 212. Encoding the downsampled data 212 produces an encoded image with a relatively low level of quality. The first device 202 may output an encoded image, for example, for transmission to a second device 204. A series of encoded images, for example, forming an encoded video, that is output for transmission to a second device 204, may be referred to as a base stream. Instead of being created in the first device 202, the encoded image may be created by an encoder that is separate from the first device 202. The encoded image may be part of a video encoded in the H.264 format. Generating processed data 213 may, for example, comprise generating successive frames of video as output from a separate encoding device, such as an H.264 video encoder. The intermediate data set for generating processed data 213 may contain the output of such an encoder, as opposed to any intermediate data generated by a separate encoder.

Producing the processed data 213 at a relatively low quality level may further include decoding the encoded image at a relatively low quality level. The decoding operation may be performed to emulate the decoding operation on the second device 204, as will be apparent below. Decoding the encoded image produces a decoded image with a relatively low level of quality. In some examples, the first device 202 decodes a relatively low quality level encoded image to create a relatively low quality level decoded image. In other examples, the first device 202 receives a decoded image at a relatively low quality level, for example, from an encoder and/or decoder that is separate from the first device 202. The encoded image may be decoded using an H.264 decoder. Decoding using a separate decoding device may comprise an incoming encoded video, such as an encoded data stream, configured to be transmitted to a remote decoding device, to a separate decoder with unknown properties implemented in conjunction with the first device 202 to generate sequential decoded video frames. Thus, the processed data 213 may comprise a frame of video data that is generated through a complex non-linear encoding and decoding process, which encoding and decoding process may involve modeling spatiotemporal correlations in accordance with a particular encoding standard such as H.264. However, since the output of any encoder is fed to a corresponding decoder, this complexity is effectively hidden from the first device 202.

In the example, generating the processed data 213 at a relatively low quality level further involves obtaining correction data based on a comparison between the subsampled data 212 and the decoded image obtained by the first device 202, for example, based on the difference between the subsampled data 212 and the decoded image. The correction data may be used to correct errors introduced during encoding and decoding of the subsampled data 212. In some examples, the first device 202 outputs the correction data, for example, for transmission to the second device 204, as well as the encoded signal. This allows the recipient to correct errors introduced during encoding and decoding of the downsampled data 212. This correction data may also be referred to as the first enhancement stream. Since the correction data can be based on the difference between the subsampled data 212 and the decoded image, it can be considered a form of residual data (eg, which is different from another set of residual data described later).

In some examples, generating processed data 213 at a relatively low level of quality further involves correcting the decoded image using the correction data. For example, the correction data as transmission output may be put into a form suitable for combining with a decoded image and then added to the decoded image. This can be done frame by frame. In other examples, rather than correcting the decoded image with the correction data, the first device 202 applies the subsampled data 212. For example, in certain cases, only encoded and then decoded data may be used, and in other cases, encoding and decoding may be replaced by another processing.

In some examples, generating the processed data 213 involves performing one or more operations other than the encoding, decoding, acquisition, and correction steps described above.

The first device 202 receives data 214 based on relatively low quality data. As noted above, such relatively low quality data may comprise processed data 213 or downsampled data 212 without such downsampled data 212 being processed at a lower level. As described above, in certain cases, the processed data 213 may comprise a reconstructed video stream (eg, from an encoding-decoding operation) that has been corrected using the correction data. In the example in FIG. 2A and 2B, data 214 is a second relatively high quality image representation, wherein the first relatively high quality image representation is input data 206. The second relatively high quality image representation can be thought of as a preliminary or predicted representation of a relatively high quality image. . In this example, the first device 202 obtains data 214 by upsampling the data at a relatively low quality level. Data 214 will hereinafter be referred to as resampled data. However, in other examples, one or more other operations may be used to generate data 214, for example, when data 212 is not obtained by downsampling input data 206. It should be noted that references to relatively high and relatively low quality levels may correspond to references to first and the second quality level, and the second quality level is higher than the first quality level. As described herein, in some cases quality levels may correspond to different spatial resolutions.

Input data 206 and resampled data 214 are used to obtain residual data 216. Residual data 216 is associated with an image. Residual data 216 may exist in the form of a set of residual elements. A residual element in the residual element set 216 may be associated with a corresponding image element in input data 206. An example of an image element is a pixel.

In this example, this residual pixel is obtained by subtracting the pixel value in the resampled data 214 from the value of the corresponding pixel in the input data 206. As such, the residual data 216 in combination with the resampled data 214 is used to reconstruct the input data 206. The residual data 216 can also called recovery data or enhancement data. In one case, residual data 216 may form part of a second enhancement stream.

The first device 202 receives configuration data related to processing the residual data 216. The configuration data indicates how the residual data 216 was processed and/or generated by the first device 202, and/or how the residual data 216 should be processed by the second device 204. The configuration data may include set of configuration parameters. Configuration data may be used to control how the second device 204 processes data and/or reconstructs input data 206 using residual data 216. Configuration data may relate to one or more characteristics of residual data 216. Different configuration data may result in different processing. performed on and/or using residual data 216. Thus, the configuration data is useful for reconstructing input data 206 using residual data 216.

In this example, the first device 202 transmits to the second device 204 data based on the downsampled data 212, data based on the residual data 216, and configuration data, allowing the second device 204 to reconstruct the input data 206.

Referring now to FIG. 2B, a second device 204 receives data 220 based (eg, generated) on the subsampled data 212. The second device 204 also receives data

- 4 045133 based on residual data 216. For example, second device 204 may receive a base stream (data 220), a first enhancement stream (any correction data), and a second enhancement stream (residual data 216). The second device 204 also receives configuration data related to processing the residual data 216. The data 220 based on the subsampled data 212 may be the subsampled data 212 itself, processed data 213, or data generated from such subsampled data 212 or processed data 213. Data based on residual data 216 may be the residual data 216 itself or data derived from such residual data 216.

In some examples, received data 220 includes processed data 213, which may contain a relatively low quality encoded image and/or correction data. In some examples, such as when the first device 202 has processed the downsampled data 212 to generate processed data 213, the second device 204 processes the received data 220 to generate processed data 222. Such processing by the second device 204 may include decoding an encoded image (for example, that forms part of the underlying encoded video stream) to create a decoded image with a relatively low level of quality. In some examples, processing by second device 204 comprises correcting the decoded image using the received correction data. Therefore, the processed data 222 may contain a frame of corrected data at a first or relatively low quality level. In some examples, the relatively low quality encoded image is decoded by a decoder that is separate from the second device 204. The relatively low quality encoded image may be decoded using an H.264 decoder.

In other examples, received data 220 contains downsampled data 212 and does not contain processed data 213. In some such examples, the second device 204 does not process the received data 220 to generate processed data 222.

To obtain the resampled data 214, the second device 204 uses relatively low quality data. As discussed above, the relatively low quality data may comprise processed data 222 or received data 220 without the second device 204 processing the relatively low quality received data 220. The resampled data 214 represents a preliminary representation of the image at a relatively high level of quality. Resampled data 214 can be generated by performing an upsampling operation on data with a relatively low quality level.

The second device 204 receives the residual data 216. To reconstruct the input data 206, the residual data 216 is used with the resampled data 214. The residual data 216 is indicative of a comparison between the input data 206 and the resampled data 214.

The second device 204 also receives configuration data related to processing the residual data 216. The configuration data may be used by the second device 204 to reconstruct the input data 206. For example, the configuration data may indicate a characteristic or property related to the residual data 216 that affects whether how the residual data 216 should be applied and/or processed, or whether the residual data 216 should be applied at all. In some examples, the configuration data includes the residual data 216.

There are several considerations regarding the use of such configuration data. One such consideration is the amount of information that is generated, stored, transmitted and/or processed. The more information is applied, the more resources can be allocated to process such information. Examples of such resources include transmission resources, storage resources, and processing resources. Compared to some known technologies, the examples described here make use of a relatively small amount of information. This may reduce the amount of data transmitted through the data network 106. The savings can be especially significant when the data relates to high quality video data, where the amount of information transferred in known systems can be especially large.

Another consideration regarding the use of such configuration data is the ability to adapt to different types of decoding devices. For example, a given encoder may transmit information to multiple decoders, each of which has different characteristics, such as processing and/or storage capabilities. It is desirable that the use of configuration data in image restoration is independent of the type and/or capabilities of the decoding device that is used to perform the restoration. Compared to some prior art technologies, the examples described herein allow different types of decoders having different performance characteristics to reliably process configuration data.

An additional consideration regarding the use of such configuration data is

- 5 045133 amount and/or complexity of processing provided in the decoder for processing configuration data.

Compared to some prior art technologies, the examples described herein reduce the amount of processing performed by the decoder to obtain and/or process configuration data.

Error handling should also be taken into account. For example, the ability of a decoder to handle potential errors when processing configuration data may be a factor in the reliability of the decoder. Compared to some prior art technologies, the examples described herein allow the decoder to handle potential errors as well as unexpected or unrecognized data. This can effectively improve the reliability of the decoder.

In fig. 3 schematically illustrates an example of a configuration message 300. A device, such as a first device 102 that includes an encoder 108, generates a configuration message 300 and outputs such a configuration message 300 for processing by an additional device, such as a second device 104 that includes a decoder 110.

Configuration message 300 contains a header portion 310 and a payload portion 320. The payload portion 320 contains configuration data 350 . Configuration data 350 relates to processing residual data, such as residual data 216, as described above. In some examples, the length of the configuration message 300 is an integer number of bytes.

The header portion 310 contains a message type parameter 312. The message type parameter 312 specifies the type of the configuration message 300. The message type specified by the message type parameter 312 may indicate how the configuration message 300 is to be processed. An example of the message type is a picture set configuration message. The picture set configuration message may correspond to multiple pictures in the picture set. Another example of a message type is an image configuration message. An image configuration message can only match a given image in an image set. Another example of a message type is an encoded data message. The encoded data message may be configured to contain encoded residual data for a given image. Another example of a message type is an encoder metadata message. The encoder metadata message may contain metadata related to one or more characteristics of the encoder. Another example of a message type is a decoder control message. The decoder control message may contain commands to control the decoder.

The message type parameter 312 indicates that the payload portion 320 has a specified payload format. The payload format of the payload portion 320 may be determined based on the message of the configuration message 300. The payload format may determine a set of configuration parameters of the configuration data 350 that are contained in the payload portion 320. In some examples, the payload format of payload portion 320 defines specific processing logic that specifies how the contents of payload portion 320 should be processed. The payload portion 320 has a format size. The format size of the payload portion 320 corresponds to the payload format of the payload portion 320. The format size of the payload portion having a predetermined payload format is predetermined or can be determined from the contents of the payload portion. For example, the format size may be determined based on the combined size of the individual configuration parameters that are determined by the payload format. The format size can be thought of as the possible size or expected size of the payload portion 320.

The header portion 310 contains a payload size parameter 314. Payload size parameter 314 specifies the actual size of the payload portion 320. The actual size of the payload portion 320 can be expressed as an integer number of bytes.

In some examples, the value of payload size parameter 314 is stored in a fixed-length element. The fixed-length element storing the value of the payload size parameter 314 may contain a predetermined number of bits in a given byte of the header portion 310. For example, the value of payload size parameter 314 may be stored in a fixed-length element that contains three bits in a one-byte header portion.

In some examples, payload size parameter 314 contains a reference to a payload size lookup table that stores possible values for the actual size of payload portion 320. For example, payload size parameter 314 may contain a value useful for obtaining the actual size of payload portion 320 from the payload size lookup table.

In some examples, the value of payload size parameter 314 is stored in a variable length element. This may be the case, for example, when the actual size of the payload portion 310 is not one of the values specified in the payload size mapping table. The actual size of the payload portion 310, determined by the payload size parameter 314, may have a variable length in bits. A variable-length element contains an integer number of bytes. IN

- 6 045133 In some examples, at least one byte of the variable length element has one or more predetermined bits configured to indicate whether the variable length element contains one or more additional bytes for such at least one specified byte. For example, a given variable length element byte may include seven bits that are configured to store a payload size parameter 314, and one bit that is used to indicate whether there is an additional byte after a given byte in the variable length element or not. As such, the payload size parameter 314 may be represented by a variable length element regardless of the bit length of the payload size parameter 314. The variable length element contains the minimum number of bytes that are usable to store the value of the payload size parameter 314.

In some examples, the data is stored in a fixed-length element, which indicates that the value of the payload size parameter 314 is stored in a variable-length element. For example, if the value of payload size parameter 314 is determined to be one of a predetermined set of values, such value may be stored in a fixed-length element and/or the fixed-length element may include a reference to a payload size lookup table. On the other hand, if it is determined that the value of the payload size parameter 314 is not one value from a predetermined set of values, a variable length element may be used to store the value, and a fixed length element may be configured to indicate, for example, using a flag, contained in a fixed-length element, such that a variable-length element is used to store the value.

In this example, the header portion 310 contains only a message type parameter 312 and a payload size parameter 314. In other examples, the header portion 310 contains one or more additional parameters. In some examples, the length of the header portion 310 is one byte. For example, the message type parameter 312 may be represented by five bits, and the payload size parameter 314 may be represented by the remaining three bits of the one-byte header portion 310. In other examples, such as where a variable length element is used to represent the actual size of the payload portion 320, the header portion 310 contains more than one byte.

The actual size of the payload portion 320 may be the same or different from the format size of the payload portion 320. Regardless of whether the actual size and the format size are the same, a payload size parameter 314 specifying the actual size is included in the header portion 310 of the configuration message 300.

The inclusion of a payload size parameter 314 that specifies the actual size of the payload portion 320 when the possible size of the payload portion 320, namely the format size, can be inferred from the message type parameter 312 may appear ineffective and/or duplicative information, especially if format size and actual size are the same. Including a payload size parameter 314 in the configuration message 300 allows for more data to be applied to the configuration message 300 compared to the case in which such a parameter is not included. However, the inclusion of the payload size parameter 314 in the configuration message 300 facilitates more efficient and reliable processing by the decoder 110 and/or improved error handling compared to the case in which such a parameter is not used, as will be described in more detail below.

In fig. 4 shows an example of a method 400 for processing configuration data. Method 400 may be implemented by a device including a decoder, such as the second device 104 described above.

Some examples accept residual data. In this example, the received residual data is the residual data 216 described above. Residual data 216 may be received from an encoder such as encoder 108. Residual data 216 is useful for reconstructing a first relatively high quality image representation 206 in a multi-level hierarchy using a second relatively high quality image representation 214. The second representation 214 is based on a relatively low quality image representation 222 in a multi-level hierarchy. In some examples, data useful for obtaining a relatively low quality representation 222 is received from encoder 108.

At position 410, a configuration message is received. In this example, the received configuration message is configuration message 300 described above. Configuration message 300 may be received from encoder 108.

At position 420, the received configuration message 300 is processed using the message type parameter 312 and the payload size parameter 314. The configuration message 300 is processed to obtain configuration data 350 stored in the payload portion 320 of the configuration message 300.

In some examples, processing of configuration message 300 includes parsing header portion 310 of configuration message 300 to obtain message type parameter 312 and payload size parameter 312. Based on the payload size parameter 314, the actual size of the payload portion 320 is determined. Based on the message type parameter 312, the type of configuration message 300 is determined. The configuration message 300 may be processed in accordance with the determined message type and the determined actual size of the payload portion 320.

Processing the configuration message 300 may include parsing the payload portion 320 to obtain configuration data 350 . In some examples, processing of the configuration message 300 includes keeping track of the number of bytes of the configuration message 300 that were read when parsing the contents of the payload portion 320. Decoder 110 may detect underread situations if there is a certain discrepancy between the number of bytes read when parsing payload portion 320 and the actual size of payload portion 320, as specified by payload size parameter 314. Incomplete reading can be a potential source of errors when processing configuration messages.

In some examples, payload size parameter 314 is used to determine the start of the next configuration message. The next configuration message may be the next sequential configuration message relative to configuration message 300 in the sequence of configuration messages received by decoder 110. The sequence of configuration messages may form part of the configuration data. Configuration message 300 and the next configuration message may correspond to the same image or different images. For example, configuration message 300 may correspond to the first image in the sequence of images, and the next configuration message may correspond to a second, different image in the sequence of images.

In some examples, processing configuration message 300 includes discarding at least a portion of the contents of payload portion 320. At least some of the content of payload portion 320 may be discarded using payload size parameter 314 . For example, decoder 110 may move to the next configuration message, where the start of the next configuration message has been determined by payload size parameter 314, without processing at least part of the contents of payload portion 320.

Thus, use of the payload size parameter 314 allows the decoder 110 to efficiently handle possible incomplete reads. Using payload size parameter 314 to determine the start position of the next message also allows decoder 110 to maintain proper alignment or synchronization with a byte stream containing multiple configuration messages. For example, even if the content and/or format of a given configuration message is unrecognized, unintended, and/or has been updated without informing decoder 110, decoder 110 is still able to determine the beginning of the next configuration message and thus remains synchronized with the byte stream. In addition, if the byte stream becomes corrupted, the decoder 110 is able to detect situations where overreading is about to occur and respond accordingly. For example, decoder 110 may interrupt processing of the current configuration message and reset the byte stream to a specific start position for the next configuration message. Thus, processing the configuration message 300 using the payload size parameter 314 improves the reliability and/or adaptability of the decoder 110 compared to the case in which such a parameter is not used, that is, the decoder 110 is able to process configuration messages that have changed and/or or were not recognized and/or were unexpected. The decoder 110 is also capable of handling potential underreads and overreads without losing synchronization with the byte stream, thereby increasing the ability of the decoder 110 to handle potential errors.

In some examples, processing of the configuration message 300 comprises parsing the first contents of the payload portion 320. The first content corresponds to the format size of the payload portion 320. The first content may contain configuration data 350. The first content may contain a first set of configuration parameters. Processing the configuration message 300 may further comprise discarding the residual contents of the payload portion 320 using the payload size parameter 314 . The residual content of the payload portion 320 contains that content of the payload portion 320 that is not contained in the first content. The remaining content contains additional configuration data. The residual content may contain an additional set of configuration parameters. The presence of residual content can be determined by detecting a discrepancy between the format size and the actual size of the payload portion 320. In other words, if it is determined that the actual size is larger than the format size, this indicates that additional content is present in the payload portion 320 in addition to the content that is provided based on the format size.

At position 430, the first representation 206 is restored using the received configuration data 350. The first representation 206 can be reconstructed using the residual data 216 in accordance with the received configuration data 350.

- 8 045133

In fig. 5 schematically illustrates an example of a configuration message 500. A device, such as a first device 102 that includes an encoder 108, may generate a configuration message 500 and output such configuration message 500 for processing by an additional device, such as a second device 104 that includes a decoder 110. Some of the elements depicted in fig. 5 are similar to the elements shown in FIG. 3, therefore, to designate similar elements, the corresponding reference numbers, increased by 200, were used.

Configuration message 500 contains a header portion 510 and a payload portion 520. The header portion 510 contains a message type parameter 512. The message type parameter 512 specifies the type of the configuration message 500. The message type specifies how the configuration message 500 should be processed. The header portion 510 contains a payload size parameter 514. Payload size parameter 514 specifies the actual size of the payload portion 520.

In this example, the payload portion 520 contains the first payload portion 522. The first payload portion 522 contains first configuration data 552 related to processing of the residual data 216. The payload portion 520 also contains a second payload portion 524. In this example, the second payload fragment 524 is positioned sequentially relative to the first payload fragment 522 in the payload portion 520. Positioning the second payload fragment 524 sequentially relative to the first payload fragment 522 may allow the decoder 110 to receive, process, and/or parse the first payload fragment 522 before the second payload fragment 524 is received, processed, and/or parsed. The second payload fragment 524 contains second configuration data 554 related to processing the residual data 216.

The first and second payload fragments 522, 524 together form the payload portion 520 of the configuration message 500. Therefore, the payload size parameter 514 indicates the combined size of the first and second payload fragments 522, 524.

In some examples, the first payload fragment 522 contains all of the first configuration data 552. In some examples, the second payload fragment 524 contains all of the second configuration data 554. As such, the first and second configuration data 552, 554 may be stored in separate portions of the payload portion 520 of the configuration message 500. Storing the first configuration data 552 in a separate portion of the payload portion 520 with respect to the second configuration data 554 facilitates efficient decoder processing of the contents of portion 520. useful data, in particular for a decoder that is configured not to use and/or process one of the first and second configuration data 552, 554.

In some examples, the header portion 510 does not specify the size of the first payload fragment 522 and/or the size of the second payload fragment 524. In some examples, the size of the first payload fragment 522 may be determined based on the format size specified by the message type parameter 512. As such, the format size specified by the message type parameter 512 may reflect the contents of the first payload fragment 522, but not the contents of the second payload fragment 524.

One or both of the first configuration data 552 and the second configuration data 554 may be useful for reconstructing the input data 206 using the residual data 216. In some examples, the second configuration data 554 includes additional configuration data. The additional configuration data is intended to complement the first configuration data 552.

The first configuration data 552 relates to a first set of one or more decoders. A first set of one or more decoders has a first performance characteristic. An example of a performance characteristic is processing capability. Second configuration data 554 relates to a second set of one or more decoders. A second set of one or more decoders has a second performance characteristic. The decoder 110 to which the configuration message 500 is transmitted may be contained in the first set or the second set. The encoder 108 that generates the configuration message 500 may not have knowledge of which set of decoders contains the decoder 110. The encoder 108 may transmit the configuration message 500 to multiple decoders through both sets of decoders.

In some examples, the first performance characteristic corresponds to a first version in which a first set of one or more decoders is configured to operate, and the second performance characteristic corresponds to a second, different version in which a second set of one or more decoders is configured to operate. The second version may be later than the first version. The first and second versions may be different versions in terms of the syntax structure that specifies how configuration messages are to be processed and/or that specifies the contents of such configuration messages. Therefore, when the syntax is updated from the first version to the second version, the second configuration data 554 is added to the configuration message that contains the first configuration data 552 from the first version. The second configuration data 554 may, for example, be appended to the first configuration data 552.

Adding a second configuration data 554 to an existing configuration message

- 9 045133 message types instead of creating a new message type to accommodate the second configuration data 554 reduces the amount of data that is stored, transmitted and/or processed compared to the case when new message types are generated. By reconfiguring existing message types instead of defining new message types, the total number of configuration messages used to convey all relevant configuration data can be reduced. Each configuration message has a corresponding header, which, for example, may be at least one byte in length. Therefore, by reducing the overall number of configuration messages, the amount of data that is stored, transmitted, and/or processed can be reduced.

In some examples, decoder 110 may not process both the first and second payload fragments 522, 524. In one case, the encoder 108 may undergo an update whereby the second configuration data 554 is included in the configuration message 500, but the decoder 110 may not undergo such an update. In other words, encoder 108 and decoder 110 may operate according to different versions of the syntax. In such examples, there may be a discrepancy in the interpretation of the format size of the payload portion 520 by the encoder 108 and the decoder 110. For example, the format size according to the encoder 108, which takes into account the additional content of the payload portion 520 and the updated payload format after a version upgrade, is different from format size according to the decoder 110, since the decoder 110 does not take into account the additional content and the updated payload format and operates in accordance with the previous version. For encoder 108, the format size is equal to the actual size of the updated payload portion 520, but for decoder 110 this is not the case. Therefore, the format size alone may not be a reliable indicator of the actual size of the payload portion 520. Thus, the inclusion of a payload size parameter 514, which specifies the actual size of the payload portion 520, facilitates more reliable processing in the decoder 110, as described in more detail below.

In fig. 6 shows an example of a method 600 for processing configuration data. Method 600 may be implemented by a device including a decoder, such as the second device 104 described above.

At position 610, configuration message 500 is received.

At position 620, configuration message 500 is processed to obtain one or both of first configuration data 552 and second configuration data 554.

In some examples, for example, if decoder 110 is in the first set of decoders, the first payload fragment 522 is parsed by decoder 110, but the second payload fragment 524 is not parsed by decoder 110. Parsing the first payload fragment 522 but not the second payload fragment 524 allows the decoder 110 to obtain the first configuration data 552 but not the second configuration data 554. In some examples, for example, if decoder 110 is in the second set of decoders, the second payload 524 is parsed by decoder 110, but the first payload 522 is not parsed by decoder 110. Parsing the second payload fragment 524, but not the first payload fragment 522, allows the decoder 110 to obtain the second configuration data 554, but not the first configuration data 552. In some examples, for example, if decoder 110 is in a second decoder set, both the first payload fragment 522 and the second payload fragment 524 are parsed to obtain both the first configuration data 552 and the second configuration data 554.

Payload size parameter 514 may be used to determine the start of the next configuration message. In some examples, for example, if decoder 110 is in the first decoder set, the contents of the second payload fragment 524 are discarded. The content of the second payload fragment 524 contains second configuration data 554. The contents of the second payload portion 524 may be discarded using the overall size of the payload portion 520 specified in the payload size parameter 514. For example, decoder 110 may read the contents of the first payload fragment 522 and then move on to the next configuration message, the beginning of which was localized by the payload size parameter 514, without reading the contents of the second payload fragment 524.

Using payload size parameter 514 to determine the start of the next message allows decoder 110 to remain synchronized with a byte stream containing multiple configuration messages. Thus, decoder 110 can process configuration message 500 and determine the start of the next configuration message regardless of whether decoder 110 is in the first set of decoders or in the second set of decoders.

At position 630, the first image representation 206 is restored at a relatively high level of quality. The first image representation 206 is reconstructed using one or both of the first configuration data 552 and the second configuration data 554 that are received at position 620. In the case where the decoder 110 receives both the first and second configuration data 552, 554, the first image representation 206 is reconstructed using both the first and second data 552, 554 con

- 10 045133 figurations. In the case where the decoder 110 processes one of the first and second payload chunks 522, 524 but not the other of the first and second payload chunks 522, 524, the first image representation 206 may be reconstructed using one of the received first and second payload data 552. , 554 configurations. As such, the use of configuration message 500 to restore the first representation 206 may be independent of the type of decoder and/or version number under which the decoder 110 is configured to operate.

In some examples, configuration message 500 performs byte-by-byte processing. In byte-byte processing, data bytes are processed one at a time. Byte-by-byte processing of configuration data can be more efficient than bit-by-bit processing of configuration data. Bit processing may involve keeping track of which bit in a given byte is the next bit to read or write. When a group of bits representing a given parameter spans a byte boundary, additional processing may be performed to read the group of bits and obtain the parameter. This processing can be reduced in the case where byte-by-byte processing is performed. In some examples, byte-by-byte processing can reduce the number of execution cycles compared to bit-by-bit processing.

In some examples, configuration data is written to memory. If configuration data is processed on a byte-by-byte basis, less processing may be required when storing configuration data in memory and/or retrieving configuration data from memory compared to the case where configuration data is processed on a bit-by-bit basis. The smallest unit of addressable memory can contain one byte. That is, information can be stored in memory byte by byte. Accordingly, if configuration data is packaged in a sequence of bytes, fewer steps may be required to store configuration data in memory and/or retrieve configuration data from memory. The amount of padding used to pack configuration data into addressable memory blocks can also be reduced by storing the configuration data in memory in byte-by-byte form.

In addition, transmission of configuration data over a network such as the Internet can be performed more efficiently if the configuration data is arranged into a sequence of bytes. Data can be transmitted over the Internet using the Transmission Control Protocol (TCP). TCP operates at the transport layer of the protocol stack. TCP receives data from the stream, packages the data into TCP segments, and uses an Internet module such as Internet Protocol (IP) to send each TCP segment over the Internet to the recipient. The data stream can be retrieved from memory. A TCP segment can contain an integer number of bytes. That is, if TCP receives data that needs to be packed into TCP segments from a byte stream, it may require less processing compared to a bit stream. The amount of padding used to pack configuration data into TCP segments can also be reduced by packing the configuration data into TCP in byte-by-byte form.

In fig. 7 shows a block diagram of an example device 700.

In the example, device 700 includes an encoder. In another example, device 700 includes a decoder.

Examples of device 700 include, but are not limited to, a mobile computer, a personal computer system, a wireless device, a base station, a telephone device, a desktop computer, a laptop computer, a notebook computer, a netbook, a mainframe computer system, a handheld computer, a workstation, a network computer, a server. applications, a storage device, a consumer electronic device such as a camera, video camera, mobile device, game console, handheld gaming device, or generally any type of computing or electronic device.

In this example, device 700 includes one or more processors 701 configured to process information and/or instructions. One or more processors 701 may include a central processing unit (CPU). One or more processors 701 are coupled to bus 702. Operations performed by one or more processors 701 may be implemented in hardware and/or software. One or more processors 701 may comprise multiple co-located processors or multiple separate processors.

In this example, device 700 includes computer-usable volatile memory 703 configured to store information and/or instructions for one or more processors 701. Computer-usable volatile memory 703 is interfaced to bus 702. Volatile memory 703 suitable for use in a computer may comprise a random access memory (RAM) device.

In this example, device 700 includes non-volatile computer-usable memory 704 configured to store information and/or instructions for one or more processors 701. Non-volatile computer-usable memory 704 is coupled to bus 702. Non-volatile memory 704 704, suitable for use in a computer, may include a read-only memory (ROM).

In this example, device 700 includes one or more data storage devices 705,

- 11 045133 designed to store information and/or commands. One or more storage devices 705 are coupled to bus 702. One or more storage devices 705 may, for example, comprise a magnetic or optical disk and a disk drive or solid-state drive (SSD).

In this example, device 700 includes one or more input/output (I/O) devices 706 configured to transmit information to and/or from one or more processors 701. One or more input/output (I/O) devices 706 are coupled to bus 702. One or more input/output (I/O) devices 706 include at least one network interface. Such at least one network interface may allow device 700 to communicate over one or more data networks. Examples of data networks include, but are not limited to, the Internet and a local area network (LAN). One or more input/output devices 706 may allow a user to provide input data to device 700 through one or more input devices (not shown). One or more input devices may include, for example, a remote control, one or more physical buttons, etc. One or more input/output devices 706 may be capable of providing information to a user through one or more output devices (not shown) . The one or more output devices may, for example, include a display screen.

Various other modules are depicted for device 700. An example would be, if present, an operating system 707, a data signal processing module 708, one or more additional modules 709 and data 710, which are shown as a single unit, or combining them with a volatile memory 703 suitable for use in a computer, a non-volatile memory 704 suitable for use in a computer, and with one or more data storage modules 705. The data signal processing module 708 may be implemented by computer program code that is stored in memory cells within computer-usable nonvolatile memory 704, computer-readable storage media in one or more data storage modules 705, and/or other tangible computer-readable media. Examples of tangible computer-readable storage media include, but are not limited to, optical media (such as a CD-ROM, DVD-ROM, or Blu-ray), flash memory module, floppy disk, hard disk, or any other media capable of storing machine-readable instructions such as firmware or microcode in at least one ROM, RAM or programmable ROM (PROM) chip or application specific integrated circuit (ASIC).

Accordingly, device 700 may include a data signal processing module 708, which may be executed by one or more processors 701. Data signal processing module 708 may be configured to include instructions for implementing at least some of the operations described herein. During operation, one or more processors 701 trigger, implement, execute, interpret, or otherwise execute instructions in the signal processing module 708.

Although at least some aspects of the examples described herein with reference to the drawings involve computer processes running on processing systems or processors, the examples described herein also extend to computer programs, such as computer programs on or in a medium made with the possibility of practical implementation of such examples. A medium can be any object or device capable of carrying a program.

It should be understood that device 700 may contain more or fewer components and/or components different from those depicted in FIG. 7.

The device 700 may be located in one location or may be distributed over multiple locations. Such locations may be local or remote.

The technologies described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware components. These may include configuring the device to implement and/or support any or all of the technologies described in this document.

It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other embodiment or any combination of any other embodiments. Moreover, without departing from the scope of the invention, which is limited by the appended claims, equivalents and modifications not described above may also be used.

-

Claims (15)

CLAIM 1. A method for generating a configuration message to enable a decoder to reconstruct a first image representation at a first quality level using configuration data, residual data and a second image representation at a first quality level, performed by a device including an encoder, the method including: obtaining residual data based on a first representation of an image at a first quality level and a second representation of an image at a first quality level, wherein the first representation and the second representation are in a multi-level hierarchy having multiple levels of quality, wherein the second representation is based on a representation of an image at a second higher quality level. low quality level in a multi-level hierarchy, wherein a second representation is obtained by upsampling an image representation with a second lower quality level, wherein the residual data is obtained by subtracting an image element in the second image representation from a corresponding image element in the first image representation, and the residual data suitable for use by a decoder to reconstruct a first representation using a second representation; obtaining configuration data related to processing of the residual data, the configuration data being suitable for use by the decoder to restore the first image representation; generating a configuration message containing a header and payload, wherein the configuration message payload contains configuration data, and the configuration message header contains: a message type parameter that specifies the message type of the configuration message and indicates that the payload has a specified payload format, in which the payload having the specified payload format has a format size; and a payload size parameter that specifies the actual size of the payload; and outputting a configuration message for processing by the decoder to enable the decoder to reconstruct the first representation using the configuration data, the residual data, and the second representation. 2. The method according to claim 1, wherein the format size and the actual size are the same. 3. The method according to claim 1, wherein the format size and the actual size are different. 4. The method according to any one of claims 1 to 3, wherein the message type specified by the message type parameter indicates how the configuration message is to be processed. 5. The method according to any one of claims 1 to 4, wherein the length of the configuration message is an integer number of bytes. 6. The method according to any one of claims 1 to 5, wherein the actual size of the payload data is an integer number of bytes. 7. The method according to any one of claims 1 to 6, wherein the value of the payload size parameter is stored in a fixed-length element. 8. The method of claim 7, wherein the value of the payload size parameter is stored in a fixed-length element that contains three bits in a one-byte header portion. 9. The method of claim 7 or 8, wherein the payload size parameter contains a reference to a payload size lookup table that stores possible values for the actual payload size. 10. The method of claim 9, wherein the specified value is applied to obtain the actual payload size from the payload size lookup table. 11. The method according to any one of claims 1 to 6, wherein the value of the payload size parameter has a variable length in bits, and wherein the value of the payload size parameter is stored in a variable length element containing an integer number of bytes. 12. The method of claim 11, wherein the variable length element contains at least one specified byte having one or more predetermined bits configured to indicate whether the variable length element contains one or more additional bytes for the at least one specified byte. 13. The method of claim 11 or 12, wherein the value is not one of a plurality of values specified in the payload size mapping table. 14. The method according to any one of claims 1 to 13, wherein the header contains only a message type parameter and a payload size parameter. 15. A method for restoring a first image representation at a first quality level using a configuration message, residual data, and a second image representation at a first quality level, performed by a device comprising a decoder, the method comprising: -