TWI757303B - Image compression method and apparatus - Google Patents

Abstract

A method of compressing image data from one or more images forming part of digital reality content, the method including obtaining pixel data from the image data, the pixel data representing an array of pixels within the one or more images; determining a position of the array of pixels within the one or more images relative to a defined position, the defined position being at least partially indicative of a point of gaze of the user; and compressing the pixel data at least partially in accordance the determined position so that a degree of compression depends on the determined position of the array of pixels.

Description

Image compression method and device

The present invention relates to a method and apparatus for compressing or decompressing video data, and in one particular example for compressing or decompressing video data, allowing transmission of video data with reduced bandwidth and less delay.

Reference in this specification to any prior publication (or derivative information thereof) or any known matter is not and should not be construed as an acknowledgement or admission or recommendation of any kind, namely, prior application (or derivative information thereof) or known matter. It constitutes part of the general knowledge in the field of business covered by this specification.

In virtual, augmented and mixed reality systems, it is common to provide a wearable display device, such as a head mounted display (HMD), which displays information to the display device based on the relative spatial position and/or orientation of the display device. wearer. Such systems operate by generating images based on information about the posture (position and orientation) of the display device, such that as the display device moves, the images are updated to reflect the new posture of the display device.

In order to avoid dizziness, it is important to minimize the time difference between the collection of posture information and the creation of the corresponding image, especially if the display device is moving rapidly. Combined with the need to produce high-resolution images so that they look as realistic as possible, this means that fairly high-end processing hardware is required. As a result, high-end existing systems often require a static desktop computer with high bandwidth and low latency connected to the display device. Therefore, current systems like the HTC Vive ^™ , Oculus Rift ^™ , and Playstation VR ^™ require a wired connection between the computer and the HMD, which is inconvenient.

While mobile solutions are available, such as the Gear VR ^™ , which is incorporated into a mobile phone to perform image processing and display within the HMD itself, processing power is limited, meaning that what can be displayed is limited, especially at image resolution and quality.

We all know that compressing image data can reduce the size of the data. This is useful in many applications, such as reducing the storage capacity required to store the image data, or reducing the bandwidth requirements associated with transmitting the image data.

JPEG uses destructive compression according to the discrete cosine transform (DCT), an arithmetic operation that converts each frame/field of the video source from the spatial (2D) domain to the frequency domain (aka transform domain). ). Perceptual models loosely based on the human mind's visual system discard high-frequency information, ie, sharp transitions in intensity and hue. In the transform domain, information is reduced by quantization. The quantized coefficients are then sequenced and loosely packed into an output bitstream.

However, this approach usually only accomplishes a limited amount of compression and requires considerable processing time, making it unsuitable for low-latency applications such as virtual or augmented reality, telepresence, etc.

In one broad form, aspects of the present invention seek to provide a method of compressing image data in one or more images forming part of digital reality content, the method comprising: obtaining pixel data from the image data, the pixel data data representing an array of pixels within the one or more images; determine the position of the array of pixels relative to a defined location within the one or more images a location, the defined location indicating at least in part a gaze point of the user; and compressing the pixel data to generate compressed image data, the pixel data being compressed at least in part according to the determined location, such that the degree of compression depends on the determined position of the pixel array.

In one embodiment, the defined position is at least one of: a measured user gaze point; an expected user gaze point; offset from a measured user gaze point; offsetting an expected location of the user's gaze point; and a location determined at least in part based on gaze data indicative of the user's gaze point, the gaze data obtained from a gaze tracking system.

In one embodiment, the method includes compressing the pixel data such that the degree of compression depends on one of the following: according to a distance from the defined point; according to a direction relative to the defined point; from the defined point points are further increased; and rendering compression is provided.

In one embodiment, the method includes: selecting one of a plurality of encoding schemes; and encoding the pixel data using the selected encoding scheme.

In one embodiment, each of the encoding schemes provides an individual degree of compression, and wherein the method includes selecting the encoding scheme based, at least in part, on at least one of: a desired degree of compression; and the pixel array of this location.

In one embodiment, the method includes: determining an encoding code indicative of the encoding scheme used; and generating compressed image data using the encoding code and encoded pixel data.

In one embodiment, the method includes using an encoding scheme to compress the pixel data by: applying a transform to the pixel data to determine a set of frequency coefficients indicative of frequency components of the pixel array; using a single bit The coding scheme is selective for at least some of these encoding the frequency components, thereby generating a set of encoded frequency coefficients; and using the encoded frequency coefficients to generate compressed image data.

In one embodiment, the bit encoding scheme defines the number of bits used to encode each of the frequency coefficients, and wherein the frequency coefficients are selectively encoded such that at least one of: at least some of the Equally encoded frequency coefficients have different numbers of bits; a smaller number of bits is used to encode frequency coefficients corresponding to higher frequencies; progressively smaller numbers of bits are used to encode frequency coefficients corresponding to gradually higher frequencies; discarding at least one frequency coefficient such that the set of coded frequency coefficients is smaller than the set of frequency coefficients; and discarding at least one frequency coefficient corresponding to a higher frequency.

In one embodiment, the method includes: selecting one of a plurality of bit encoding schemes; and encoding the frequency coefficients according to the selected bit encoding scheme.

In one embodiment, each of the plurality of bit encoding schemes selectively encodes different frequency coefficients with different numbers of bits to provide different degrees of compression.

In one embodiment, the bit encoding scheme is selected based, at least in part, on at least one of: a desired degree of compression; and the position of the pixel array.

In one embodiment, the frequency components are arranged in a plurality of levels, and wherein each bit encoding scheme defines a respective bit to be used to encode the frequency coefficients in each of the plurality of levels arity.

In one embodiment, the array is an NxN array of pixels, resulting in 2N - 1 levels of frequency components.

In one embodiment, the method includes applying a scaling factor to at least some of the frequency coefficients, thus encoding the scaled frequency coefficients, the scaling factor is used to reduce the intensity of each frequency coefficient, and at least one of the following: apply a different scaling factor to at least some of the frequency coefficients; apply the same scaling factor to each frequency coefficient; and apply a different scaling factor The scaling factor of is applied to the frequency coefficients within a different channel.

In one embodiment, the image data defines a plurality of channels, and wherein the method includes selectively encoding frequency coefficients for each channel.

In one embodiment, the pixel data defines RGB channels, and wherein the method includes: converting the RGB channels to YCbCr channels; and converting the YCbCr channels.

In one embodiment, the method includes at least one of: selectively encoding more frequency coefficients for the Y channel than the Cb or Cr channels; selectively encoding frequency coefficients simultaneously for the YCbCr channels; and selectively The frequency coefficients are encoded for the CbCr channels and the Y channel is used.

In one embodiment, the transform is a two-dimensional (2-D) discrete cosine transform.

In one embodiment, the method includes obtaining pixel data from image data using: buffering image data corresponding to the next n-1 rows of pixels of the image; buffering image data of the next n pixels of the next row of pixels; obtain pixel data for the next n x n block of pixels from the buffered image data; repeat steps b) and c) until pixel data has been obtained from all the n rows of pixels; and repeat steps a) to d), until pixel data has been obtained from each column of pixels in the image.

In one embodiment, n is selected according to at least one of: a selected bit encoding scheme; a desired degree of compression; and the position of the pixel array.

In a specific embodiment, the method includes simultaneously selecting an encoding frequency system and generating compressed image data at least in part by parallel-to-sequential byte encoding.

In one broad form, aspects of the present invention seek to provide a method of decompressing compressed image data in one or more images forming part of digital reality content, the method comprising: obtaining the compressed image data, The compressed image data represents an array of pixels within the one or more images, and is compressed based at least in part on a position of the array of pixels within the one or more images relative to a defined position, the compressed image Defining a location indicates, at least in part, a gaze point of the user; and decompressing the compressed image data based at least in part on the determined location.

In one embodiment, the defined position is at least one of: a measured user gaze point; an expected user gaze point; offset from a measured user gaze point; offsetting an expected location of the user's gaze point; and a location determined at least in part based on gaze data indicative of the user's gaze point, the gaze data obtained from a gaze tracking system.

In one embodiment, the method includes: selecting one of a plurality of decoding schemes; and decoding the pixel data using the selected decoding scheme.

In one embodiment, the method includes selecting the decoding scheme based, at least in part, on at least one of: a desired degree of compression; a position in the pixel array; and an encoding code indicating one of the encoding schemes used, the The encoding code is determined by the compressed data image.

In one embodiment, the method includes decompressing the compressed pixel data using a decoding scheme according to a one-bit encoding scheme in which the number of bits used in each encoded frequency coefficient is defined, determine a set of encoded video data from the compressed image data rate coefficients; performing bit-wise decoding of the coded frequency coefficients in accordance with the bit-coding scheme, thereby generating a set of frequency coefficients, wherein at least one frequency coefficient is generated, such that the set of coded frequency coefficients is smaller than the set of frequency coefficients; and An inverse transformation is applied to the set of frequency coefficients to determine pixel data representing an array of pixels within the one or more images.

In one embodiment, the bit-coding scheme defines the number of bits used to encode each of the frequency coefficients, the bit-coding scheme uses a smaller number of bits to encode corresponding to higher frequencies frequency coefficients, and wherein the method includes generating at least some of the frequency coefficients corresponding to the higher frequency domain.

In one embodiment, the method includes: selecting one of a plurality of bit encoding schemes; and decoding the encoded frequency coefficients according to the selected bit encoding scheme.

In one embodiment, each of the plurality of bit encoding schemes selectively encodes different frequency coefficients with different numbers of bits to provide a different degree of compression.

In one embodiment, the bit encoding scheme is selected based at least in part on at least one of: an encoding code; the bit encoding scheme used to generate the compressed image data; and the position of the pixel array.

In one embodiment, the frequency components are arranged in a plurality of levels, and wherein each bit encoding scheme defines a respective bit to be used to encode the frequency coefficients in each of the plurality of levels arity.

In one embodiment, the array is an NxN array of pixels, resulting in 2N - 1 levels of frequency components.

In one embodiment, the method includes applying a scaling factor to at least some of the frequency coefficients, such that decoding the scaled encoded frequency coefficients, the scale A scaling factor is used to increase the strength of each frequency coefficient, and at least one of the following: apply a different scaling factor to at least some of the coded frequency coefficients; apply the same scaling factor to each coded frequency coefficient ; and applying a different scaling factor to the encoded frequency coefficients in a different channel.

In one embodiment, the image data defines a plurality of channels, and wherein the method includes selectively decoding the encoded frequency coefficients for each channel.

In one embodiment, the compressed image data defines YCbCr channels, and wherein the method includes: performing an inverse conversion of the YCbCr channels; and converting the converted YCbCr channels to RGB channels.

In one embodiment, the method includes at least one of: generating more frequency coefficients for the Cb or Cr channel than the Y channel; simultaneously decoding the encoded YCbCr channels; and decoding the CbCr channels and converting the Etc. the encoded CbCr channel and the Y channel are converted to RGB channels.

In one embodiment, the inverse transform is an inverse two-dimensional (2-D) discrete cosine transform.

In one embodiment, the method includes: decoding the compressed video data at least in part by sequential-to-parallel byte decoding; and simultaneously selectively decoding the frequency coefficients.

In one embodiment, the degree of compression desired is determined based on at least one of: the location of the pixel array; the transmission bandwidth of a communication link used to transmit the compressed image data; the transmission bandwidth used to transmit the compressed image Data transmission quality of a communication link; movement of a display device; image display requirements; a target display resolution; a processed channel; and an error matrix.

In one embodiment, the digital reality is at least one of: augmented reality; virtual reality; mixed reality; and telepresence.

In one embodiment, the method is for transmitting the image data from a computing device to a wearable digital reality headset via at least one of: a communication network; and a wireless communication link.

In one broad form, aspects of the present invention seek to provide an apparatus for compressing image data in one or more images forming part of digital reality content, the apparatus comprising at least one electronic encoder processing device, the at least one Electronic encoder processing device: obtains pixel data from the image data, the pixel data data represents a pixel array within the one or more images; determines the pixel array within the one or more images relative to a defined a location of a location, the defined location indicating at least in part a gaze point of the user; and compressing the pixel data at least in part according to the determined location, the extent of such compression being dependent on the determined location of the pixel array.

In one broad form, aspects of the present invention seek to provide an apparatus for decompressing compressed image data in one or more images forming part of digital reality content, the apparatus comprising at least one electronic decoder processing device, The at least one electronic decoder processing device: obtains compressed image data, the compressed image data representing an array of pixels within the one or more images, and is based, at least in part, on relative to the one or more images compressing a location of the pixel array at a defined location, the defined location indicating at least in part the user's gaze point; and decompressing the compressed image data based at least in part on the determined location.

In one broad form, aspects of the present invention seek to provide a method of compressing image data representing one or more images, the method comprising: obtaining pixels from the image data data representing an array of pixels within the one or more images; applying a transformation to the pixel data to determine a set of frequency coefficients indicative of frequency components of the pixel array; selecting using a one-bit encoding scheme to selectively encode at least some of the frequency coefficients, thereby producing a set of encoded frequency coefficients, wherein the bit encoding scheme defines the number of bits used to encode each of the frequency coefficients, such that the frequency coefficients are selected When encoding: encoding at least some of the encoded frequency coefficients using different numbers of bits; and discarding at least one frequency coefficient such that the group of encoded frequency coefficients is smaller than the group of frequency coefficients; and using the encoded frequency coefficients to generate compression image data.

In one embodiment, the frequency coefficients are selectively encoded such that at least one of the following: defines the use of a number of bits to encode each of the frequency coefficients, independent of the value of the respective frequency coefficient; a smaller number of bits for encoding frequency coefficients corresponding to higher frequencies; gradually fewer bits are used for encoding frequency coefficients corresponding to gradually higher frequencies; discarding at least one frequency coefficient corresponding to higher frequencies.

In one embodiment, the method includes applying a scaling factor to at least some of the frequency coefficients, thus encoding the scaled frequency coefficients, and wherein at least one of the following: applying a different scaling factor to at least some of the frequency coefficients the same scaling factor is applied to each frequency coefficient; and the scaling factor is used to reduce the intensity of each of the frequency coefficients.

In one embodiment, the method includes: selecting one of a plurality of encoding schemes; and encoding the pixel data using the selected encoding scheme.

In one embodiment, each of the encoding schemes provides a respective degree of compression, and wherein the method includes selecting the encoding based, at least in part, on at least one of Scheme: a desired degree of compression; and the position of the pixel array.

In one embodiment, the method includes selectively encoding the frequency coefficients according to at least one of: selection rules; a desired degree of compression; and a location of the pixel array within the one or more images.

In one embodiment, the method includes: selecting one of a plurality of bit encoding schemes; and encoding the frequency coefficients according to the selected bit encoding scheme.

In one embodiment, each of the plurality of bit encoding schemes selectively encodes different frequency coefficients with different numbers of bits to provide different degrees of compression.

In one embodiment, the bit encoding scheme is selected based, at least in part, on at least one of: selection rules; a desired degree of compression; and the location of the pixel array.

In one embodiment, the method includes selecting the bit decoding scheme based on at least one of: a transmission bandwidth of a communication link used to transmit the compressed video data; a transmission bandwidth used to transmit the compressed video data the transmission quality of the communication link; the movement of a display device; the demand for image display; a target display resolution; a channel processed; a position of the pixel array within the one or more images; the one or more A position within an image of the pixel array relative to an observer's gaze point of the one or more images; and an error matrix.

In one embodiment, the frequency components are arranged in a plurality of levels, and wherein each bit encoding scheme defines a respective bit to be used to encode the frequency coefficients in each of the plurality of levels arity.

In one embodiment, the array is an NxN array of pixels, resulting in 2N - 1 levels of frequency components.

In one embodiment, the method includes: determining the one or more images A gaze point of an observer; the frequency coefficients are selectively encoded at least in part according to the gaze point.

In one embodiment, the method includes: determining a distance between the gaze point and a position of the pixel array in the one or more images; and selectively encoding frequency coefficients according to the distance, such that the larger distance is Fewer frequency coefficients are encoded.

In one embodiment, the image data defines a plurality of channels, and wherein the method includes selectively encoding frequency coefficients for each channel.

In one embodiment, the pixel data defines RGB channels, and wherein the method includes: converting the RGB channels to YCbCr channels; and converting the YCbCr channels.

In one embodiment, the method includes at least one of: selectively encoding more frequency coefficients for the Y channel than the Cb or Cr channels; selectively encoding more frequency coefficients simultaneously for the YCbCr channels; and The compressed image data is generated by: encoding the CbCr channels; and using the Y channel.

In one embodiment, the transform is a two-dimensional (2-D) discrete cosine transform.

In one embodiment, the method includes obtaining the pixel data from a video feed.

In one embodiment, the method includes obtaining pixel data from image data using: buffering image data corresponding to the next n -1 rows of pixels of the image; buffering image data of the next n pixels of the next row of pixels; obtain pixel data for the next n x n block of pixels from the buffered image data; repeat steps b) and c) until pixel data has been obtained from all the n rows of pixels; and repeat steps a) to d), until pixel data has been obtained from each column of pixels in the image.

In one embodiment, n is selected according to at least one of: a selection rule; a selected bit-coding scheme; and the position of the pixel array.

In one embodiment, the method includes simultaneously selecting encoding frequency coefficients; and generating compressed image data at least in part by parallel-to-sequential byte encoding.

In one broad form, aspects of the present invention seek to provide an apparatus for compressing image data representing one or more images, the apparatus comprising at least one electronic encoder processing means, the at least one electronic encoder processing means: from The image data obtains pixel data representing an array of pixels within the one or more images; applies a transformation to the pixel data to determine a set of frequency coefficients indicative of frequency components of the pixel array; uses a A bit-coding scheme selectively encodes at least some of the frequency coefficients, thereby producing a set of coded frequency coefficients, wherein the bit-coding scheme defines the number of bits used to encode the frequency coefficients, and wherein the When frequency coefficients are selectively encoded: at least some of the encoded frequency coefficients have different numbers of bits; discard at least one frequency coefficient so that the group of encoded frequency coefficients is smaller than the group of frequency coefficients; and use the encoded frequency coefficients to generate Compressed image data.

In one embodiment, the apparatus includes: an encoder input buffer for receiving the image data; and an encoder output buffer for storing compressed image data.

In one embodiment, the apparatus includes an encoder input buffer that: buffers image data corresponding to the next n -1 rows of pixels in the image; buffers the next n of pixels in the next row pixel image data, allowing the at least one encoder processing device to obtain pixel data for the next n x n pixel block from the buffered image data; repeat step b) until pixel data has been obtained from all the n rows of pixels ; and repeating steps a) and b) until pixel data has been obtained from each column of pixels in the image.

In one embodiment, the apparatus includes an encoder transmitter that transmits the image data from the encoder output buffer.

In one embodiment, the at least one encoder processing device includes: a field programmable gate array; a dedicated integrated circuit; and a graphics processing unit.

In one embodiment, the pixel data defines a plurality of channels, and wherein the apparatus includes at least one of a respective processing device for each channel and a parallel processing device for processing each channel simultaneously.

In one embodiment, the pixel data defines RGB channels, and wherein the apparatus: converts the RGB channels to YCbCr channels; and selectively encodes the YCbCr channels using processing means.

In one embodiment, the pixel data defines RGB channels, and wherein the apparatus: converts the RGB channels to CbCr channels using a YCbCr processing device; decodes the CbCr channels using at least one respective processing device; and The Y channel is converted from the YCbCr processing device to an output buffer using a delay block.

In one embodiment, the apparatus includes an encoder in wireless communication with a decoder, allowing image data to be converted between the encoder and the decoder as compressed image data.

In one embodiment, the encoder is at least one and a portion thereof coupled to a suitable programming processing system.

In one embodiment, the decoder is coupled to at least one and a portion of a wearable display device.

In one embodiment, the encoder and decoder communicate to exchange at least The following one: compressed image data; movement data indicating movement of a display device; control data used at least in part to control the display device; input data indicating user input commands; gaze data indicating a gaze point of an observer ; and sensor data from sensors associated with a wearable display device.

In one broad form, aspects of the present invention seek to provide a method of decompressing compressed image data representing one or more images, the method comprising: obtaining the compressed image data; each encoded frequency as defined A one-bit encoding scheme of the number of bits used in the coefficients, and a set of encoded frequency coefficients is determined from the compressed image data; bit-wise decoding of the encoded frequency coefficients is performed according to the bit encoding scheme, thereby generating a a set of frequency coefficients, wherein at least one frequency coefficient is generated, such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and applying an inverse transformation to the set of frequency coefficients to determine an array representing a pixel within the one or more images pixel data.

In one embodiment, the method includes: selecting one of a plurality of decoding schemes; and decoding the pixel data using the selected decoding scheme.

In one embodiment, the method includes selecting the decoding scheme based, at least in part, on at least one of: selection rules; a desired degree of compression; a position in the pixel array; and an encoding indicating one of the encoding schemes used code, the encoding code is determined by the compressed data image.

In one embodiment, the bit encoding scheme uses a smaller number of bits to encode frequency coefficients corresponding to higher frequencies, and wherein the method includes generating at least some of the frequency coefficients corresponding to the higher frequency domain.

In a specific embodiment, the method includes applying a scaling factor to at least some of these frequency coefficients, such that the scaled frequency coefficients are converted and at least one of the following: apply a different scaling factor to at least some of the encoded frequency coefficients; apply the same scaling factor to each encoded frequency coefficients; and the scaling factor is used to increase the strength of each of the encoded frequency coefficients.

In one embodiment, the method includes: selecting one of a plurality of bit encoding schemes; and decoding the encoded frequency coefficients according to the selected bit encoding scheme.

In one embodiment, each of the plurality of bit encoding schemes selectively encodes different frequency coefficients with different numbers of bits to provide different degrees of compression.

In one embodiment, the bit encoding scheme is selected based at least in part on at least one of: an encoding code; selection rules; the bit encoding scheme used to generate the compressed image data; the location.

In one embodiment, the selection rule is based on at least one of the following: transmission bandwidth of a communication link used to transmit the compressed image data; transmission quality of service of a communication link used to transmit the compressed image data; movement of a display device; image display requirements; a target display resolution; a channel processed; a position of the pixel array within the one or more images; the pixel array within the one or more images a position of an observer's gaze point relative to the one or more images; and an error matrix.

In one embodiment, the method includes: determining a gaze point of an observer of the one or more images; and selectively decoding encoded frequency coefficients based at least in part on the gaze point.

In one embodiment, the method includes: determining a distance between the gaze point and a position of the pixel array within the one or more images; and according to the distance the encoded The frequency coefficients are selectively decoded, such that larger distances produce more frequency coefficients.

In one embodiment, the frequency components are arranged in a plurality of levels, and wherein each bit encoding scheme defines a respective bit to be used to encode the frequency coefficients in each of the plurality of levels arity.

In one embodiment, the array is an NxN array of pixels, resulting in 2N - 1 levels of frequency components.

In one embodiment, the image data defines a plurality of channels, and wherein the method includes selectively decoding the encoded frequency coefficients for each channel.

In one embodiment, the compressed image data defines YCbCr channels, and wherein the method includes: performing an inverse conversion of the YCbCr channels; and converting the converted YCbCr channels to RGB channels.

In one embodiment, the method includes at least one of: generating more frequency coefficients for the Cb or Cr channel than the Y channel; simultaneously decoding the encoded YCbCr channels; decoding the CbCr channels, and converting the The decoded CbCr channel and the Y channel are converted to RGB channels.

In one embodiment, the inverse transform is an inverse two-dimensional (2-D) discrete cosine transform.

In one embodiment, the method includes using the pixel data to generate a video feed.

In one embodiment, the method includes: decoding the compressed video data at least in part by sequential-to-parallel byte decoding; and simultaneously selectively decoding the frequency coefficients.

In one embodiment, the digital reality is at least one of: augmented reality; virtual reality; and mixed reality.

In one embodiment, the method is for displaying the image data in a wearable digital reality headset utilizing the compressed image data received from a computing device through at least one of: a communication network; and a wireless communication link.

In a specific embodiment, the method is used for at least one of: transmitting virtual reality video material; and wirelessly transmitting virtual reality video material.

In one broad form, aspects of the present invention seek to provide an apparatus for decompressing compressed image data representing one or more images, the apparatus comprising at least one electronic decoder processing means, the at least one electronic decoder processing The device: obtains compressed image data; determines a set of coded frequency coefficients from the compressed image data according to a one-bit coding scheme that defines the number of bits used in each coded frequency coefficient; encodes according to the bits a scheme to perform bit-wise decoding of the coded frequency coefficients, thereby generating a set of frequency coefficients, wherein at least one frequency coefficient is generated, such that the set of coded frequency coefficients is smaller than the set of frequency coefficients; and applying an inverse to the set of frequency coefficients transform to determine pixel data representing an array of pixels within the one or more images.

In one embodiment, the apparatus includes: a decoder input buffer for receiving the compressed image data; and a decoder output buffer for storing the image data.

In one embodiment, the apparatus includes a decoder transceiver that receives the compressed image data and provides the compressed image data to the input buffer.

In one embodiment, the at least one decoder processing device includes: a field programmable gate array; a dedicated integrated circuit; and a graphics processing unit.

In one embodiment, the compressed image data defines a plurality of channels, And wherein the apparatus includes at least one of a respective processing device for each channel and a parallel processing device for processing each channel simultaneously.

In one embodiment, the compressed image data defines YCbCr channels, and wherein the apparatus: decodes the CbCr channels using at least one processing device; and converts the decoded YCbCr channels to RGB channels.

In one embodiment, the compressed image data defines YCbCr channels, and wherein the apparatus: decodes the CbCr channels using processing means; converts the decoded CbCr channels and the Y channel using an RGB processing means RGB channel; and converting the Y channel from a decoder input buffer to the RGB processing device using a delay block.

In one embodiment, the apparatus includes a decoder in wireless communication with an encoder, allowing image data to be transmitted between the encoder and the decoder as compressed image data.

In one embodiment, the decoder is at least one and a portion thereof coupled to a suitable programming computer system.

In one embodiment, the decoder is coupled to at least one and a portion of a wearable display device.

In one embodiment, the decoder communicates with the encoder to exchange at least one of: compressed video data; movement data indicating movement of a display device; control data used at least in part to control the display device; input data for the user to input commands; gaze data indicating a gaze point of an observer; and sensor data from sensors associated with a wearable display device.

In a broad form, aspects of the present invention seek to provide a A method of compressing image data of one or more images, the method comprising: obtaining pixel data from the image data, the pixel data representing an array of pixels within the one or more images; determining an encoding scheme; using the encoding scheme encoding the pixel data; determining an encoding code indicating the encoding scheme used; and generating compressed image data using the encoding code and the encoded pixel data.

In one embodiment, the method includes determining the encoding scheme by at least one of: based on an image type of the image data; based on an encoding scheme indication received from an image data source; by analyzing at least the image data with one of the pixel data; and according to a compression requirement comprising at least one of: a compression amount; a resultant image quality; and a compression delay.

In one embodiment, the method includes analyzing the pixel data to determine whether the pixel array is at least one of: a gradient; a border; and a single color.

In one embodiment, the method includes at least one of the following: if the pixel array is a solid color, replacing the pixel array with an encoded code representing the solid color; if the pixel array is a gradient, the method includes using The pixel data is encoded according to a method according to another aspect of the present invention; and the pixel data is decoded using a method according to another aspect of the present invention.

In one broad form, aspects of the present invention seek to provide an apparatus for compressing image data representing one or more images, the apparatus comprising at least one electronic encoder processing means, the at least one electronic encoder processing means: from the image data obtains pixel data representing an array of pixels within the one or more images; determines an encoding scheme; encodes the pixel data using the encoding scheme; determines an encoding indicating the encoding scheme used code; and generating compressed image data using the encoded frequency coefficients.

In one broad form, aspects of the present invention seek to provide a method of decompressing compressed image data representing one or more images, the method comprising: obtaining compressed image data; determining from the compressed image data an encoding code; determining an encoding scheme using the encoding code; and decoding the at least a portion of the compressed image data using the encoding scheme to determine pixel data representing an array of pixels within the one or more images.

In one embodiment, the method includes at least one of: replacing an encoded code with a solid-color pixel array; using a method according to another aspect of the present invention to decode compressed image data into a gradient of pixel array; and decoding the compressed image data using a method according to another aspect of the present invention.

In one broad form, aspects of the present invention seek to provide an apparatus for decompressing compressed image data representing one or more images, the apparatus comprising at least one electronic decoder processing means, the at least one electronic decoder processing means: obtaining compressed image data; determining an encoding code from the compressed image data; determining an encoding scheme using the encoding code; and decoding the compressed image data using the encoding scheme to determine representations of the one or more Pixel data for a pixel array within an image.

It should be appreciated that the broad forms of the invention and their respective features may be used in combination, interchangeably and/or independently and are not intended to be limited to reference to a single broad form.

100-170‧‧‧Steps

200‧‧‧Display video equipment on wearable devices

210‧‧‧Processing system

220‧‧‧Encoder

230‧‧‧Decoder

240‧‧‧HMD

250‧‧‧Controller

260‧‧‧Wireless Communication Link

300‧‧‧Virtual reality systems incorporated for Equipment for compressing and decompressing video data

310‧‧‧Processing system

311‧‧‧Microprocessors

312‧‧‧Memory

313‧‧‧Input/Output Devices

314‧‧‧External interface

315‧‧‧Bus

320‧‧‧Encoder

321‧‧‧Encoder input buffer

322‧‧‧Encoder processing device

323‧‧‧Encoder output buffer

324‧‧‧Transceiver

325‧‧‧Individual data buffer

330‧‧‧Decoder

331‧‧‧Decoder input buffer

332‧‧‧Decoder Processing Device

333‧‧‧Decoder output buffer

334‧‧‧Transceivers

335‧‧‧Individual data buffer

340‧‧‧Display Devices

341‧‧‧Microprocessors

342‧‧‧Memory

343‧‧‧Input/Output Devices

344‧‧‧Sensor

345‧‧‧Display

346‧‧‧External interface

347‧‧‧Bus

360‧‧‧Short range radio communication

400-448‧‧‧Steps

531-533‧‧‧Frequency coefficient

541‧‧‧Luminance Channels

542‧‧‧chroma channels

543‧‧‧chroma channel

600-630‧‧‧Steps

701‧‧‧Viewable part

702‧‧‧Unviewable part

800-860‧‧‧Steps

900-928‧‧‧Steps

1000-1040‧‧‧Steps

Please refer to the accompanying drawings to illustrate examples of the present invention, wherein: the first is a flowchart of an example method for compressing and subsequently decompressing image data; the second A is for displaying images on a wearable device one of the devices The first example is a schematic diagram; the second B is a second example of an apparatus for displaying images on a wearable device; the third diagram is a virtual reality system incorporating equipment for compressing and decompressing image data Figure 4 is a schematic diagram of a specific example; Figures 4 A to D are flowcharts of a specific example of a method for compressing and subsequently decompressing image data; Figure 5 is a schematic diagram illustrating an aspect of the encoding process; Figure 6 is a A flowchart of an example method of selecting a one-bit encoding scheme; Figure 7 is a schematic diagram of an example of an image to be encoded; Figure 8 is another example of an encoding/decoding scheme for compressing image data and decompressing compressed image data Flow chart of an example; Figures 9A and 9B are flow charts of a specific example of a method for compressing and subsequently decompressing image data using a selected encoding/decoding scheme; and Figure 10 is a flow chart for compression and a flowchart of a further example of a method for subsequent decompression of image data.

An example of a method for compressing and decompressing image data will now be described with reference to the first figure.

For purposes of illustration, it is assumed that the processing is performed, at least in part, using one or more electronic processing devices. In one example, separate processing devices are used to compress and decompress the image data, allowing compressed image data to be transmitted between the two processing devices, although this is not required and And alternatively the same processing device can be used to compress and decompress the image data.

The processing device may form part of a respective processing system, such as a computer system, a computer server, a client device including a mobile phone, a portable computer, a display device such as a wearable or head-mounted display, or otherwise It may be in the form of a stand-alone module attached to such a device.

The image data typically represents one or more images, and in one example, a series of images to be displayed on respective display devices. As will be understood from the description below, in one particular example, the image data is suitable for remote display of a series of images for a source, such as a virtual or augmented reality in which the images are displayed on a wearable display In graphics applications, and/or in telepresence applications where images are displayed from a remotely controllable system such as a drone mounted camera.

In this example, at step 100, pixel data is obtained from image data, the pixel data representing an array of pixels within the one or more images. The pixel data can be obtained in any suitable manner, depending on the format of the image data. In one example, this is accomplished by simply selecting a particular sequence of bytes within the image data. The pixel array typically corresponds to a number of pixels, such as an 8x8 pixel block within one of the images, although other pixel arrays may be used.

At step 110, a transformation is applied to the pixel data to determine a set of frequency coefficients indicative of frequency components of the pixel array. Therefore, the transform is usually a frequency transform, such as a Fourier transform, etc., and in one example a 2D DCT (discrete cosine transform). The transformation can be applied in any suitable manner, such as using known transformation techniques, but in one example is performed in a highly parallel manner, thereby reducing processing time.

At step 120, at least some of the frequencies are encoded using a one-bit encoding scheme number-selective encoding, thereby producing a set of encoded frequency coefficients. The bit encoding scheme defines the number of bits to be used to encode each frequency coefficient, after the frequency coefficients are selectively encoded, at least some of the encoded frequency coefficients have different numbers of bits, and at least one frequency coefficient is discarded so that the set of coded frequency coefficients is smaller than the set of frequency coefficients.

This process may be accomplished in any suitable manner, and may include discarding some of the frequency coefficients and then encoding the remaining frequency coefficients using a different number of bits, thereby reducing the number of bits required to encode the frequency coefficients. Alternatively, the process should include encoding some of the frequency coefficients with zero bits, thereby effectively discarding the respective frequency coefficients as part of the encoding step.

The specific frequency components that are discarded will vary with the preferred implementation, generally higher frequency components are discarded because they are less intense and because they correspond to sharp transitions within the image, meaning they have a significant impact on the overall image quality. Contribute less. This allows higher frequency component coefficients to be discarded without significantly adversely affecting the perceived image quality. In addition to discarding frequency components corresponding to higher frequencies, the process may encode frequency coefficients of higher frequency components with fewer bits, thereby reducing the total number of bits required to encode the frequency coefficients.

Similarly, when encoding the frequency coefficients with different numbers of bits, it is performed independently of the actual value of the frequency coefficient, but rather based on knowledge of the expected strength of the frequency coefficient. For example, frequency coefficients at lower frequencies are generally stronger and therefore encoded with more bits, while frequency coefficients at higher frequencies are typically less intense and therefore encoded with fewer bits. This allows the frequency coefficient values to be encoded without loss of information.

Once the encoding has been performed, the encoded frequency coefficients may be used to generate compressed image data at step 130, for example, this may be performed by creating a stream of bits containing the sequence of encoded frequency coefficients, optionally with additional information, such as flags or other markings to identify The start of new images and more.

Thus, the above process allows the creation of compressed frequency coefficients by using a one-bit encoding scheme that discards at least some of the frequency coefficients and encodes the remaining coefficients using a different number of bits, eg selectively encoding frequency coefficients according to the strength of the frequency coefficients video material. In this way, smaller intensity coefficients can be encoded using a smaller number of bits without any loss of information.

It should be noted that this approach should be contrasted with code substitution techniques such as Huffman encoding, where values are substituted for shorter codes. Instead, in this example, the values are still encoded, albeit using the number of bits appropriate to the expected strength of the value, so if the frequency coefficient value is not expected to exceed seven, then this can be encoded as a three-bit character, so Six will be encoded as "110" instead of using the default octet "00000110". In contrast, if the value of the frequency coefficient is expected to be as high as sixty-three, six-bit characters should be used, eg, twenty should be encoded as "010100". In the event that the value is higher than the available number of bits, the maximum value available for the defined number of bits should be used, resulting in a loss of precision in the resulting compressed image data.

As such, the bit encoding scheme uses information about the expected size of the frequency coefficient values in order to define the number of bits that should be used. A less aggressive bit-coding scheme will use more bits, resulting in less compression, but at higher resolution, while a more aggressive bit-coding scheme will use fewer bits and therefore provide more compression, But the resolution is lower.

In any event, by using a one-bit encoding scheme, which defines the number of bits used to encode each frequency coefficient, this allows the same scheme to be used when decompressing the compressed image data, which in turn allows correct decompression to be performed , while allowing the bit-coding scheme used to be set to optimize the compression for the current situation.

In this regard, at step 140, according to the bit encoding scheme, from the Compressed image data determines a set of encoded frequency coefficients. In particular, information about the number of bits used to encode each frequency coefficient allows dividing the received compressed image data into the encoded frequency coefficients by selecting the next number of bits that make up the next frequency coefficient .

At step 150, selective bit decoding of the encoded frequency coefficients is performed according to the bit encoding scheme, thereby generating a set of frequency coefficients. In this regard, execution is performed to convert each encoded frequency coefficient into a frequency coefficient and additionally generate frequency coefficients to be discarded during the encoding process. In particular, it is usually performed to generate frequency coefficients with null values, thereby again establishing a whole set of frequency coefficients.

Next, an inverse transformation is applied to the set of frequency coefficients to determine pixel data representing a pixel array within the one or more images. In particular, it is usually in the form of an inverse frequency transform, such as an inverse Fourier transform, 2D DCT, and the like.

Thus, the above-described process allows the encoding of image data by selectively encoding frequency coefficients using a one-bit encoding scheme, and then decoding the encoded frequency coefficients using the same bit-encoding scheme. Furthermore, the bit encoding scheme used can be adapted and can depend on a wide range of criteria, such as the nature of the encoded image data, the particular channel being encoded, and so on. This allows applying the bitwise encoding scheme, whereby the maximum amount of compression can be achieved.

In addition to the above advantages, the scheme can be implemented in a highly parallel manner, eg allowing each frequency coefficient to be encoded simultaneously. This in turn enables the processing to be performed quickly, thereby reducing latency, which is important in many applications, such as virtual reality applications, where an image is created in response to movement of a display device and must be quickly transmitted to the display device to show.

Some further features will be described at this point.

In one example, the bit encoding scheme uses a smaller number of bits to encode The frequency coefficients corresponding to the higher frequencies, since these higher frequency components have less intensity, which means that fewer bits are required to accurately encode the higher frequency coefficients than the lower frequencies. In one example, progressively smaller numbers of bits are used to encode frequency coefficients corresponding to progressively higher frequencies. In this case, the frequency coefficients at successively higher frequencies should have the same or lower number of bits than the frequency coefficients of a previous lower frequency. Similarly, the method may include discarding at least some of the frequency coefficients corresponding to higher frequencies, which tend to have less impact on the quality of the image seen. It should also be appreciated that for some of these lower frequency coefficients, all bits required to encode the coefficient may be reserved, in some examples exceeding 8 bits.

In one example, the method includes applying a scaling factor to at least some of the frequency coefficients, thus encoding the scaled frequency coefficients. In this regard, scaling is used to reduce the intensity of the frequency coefficients so that the coefficients can be encoded using a smaller number of bits. Similar scaling factors may be applied when performing decompression, thereby scaling the respective frequency components back to their original intensities. During this process, numerical rounding is typically performed so that the scaled frequency components are integer-valued, or have a limited amount of significant data, thereby minimizing the number of bits used to encode the coefficients. It should also be appreciated that, when implemented, the accuracy of the reconstructed frequency components is reduced, but this effect has a negligible impact on the quality of the resulting image.

In one example, the same scaling factor is applied to each frequency coefficient. This is particularly advantageous as this reduces the computational burden when performing this scaling. In particular, this allows a single scaling factor to be read from memory, such as a scratchpad, or to be hard-coded within a logical configuration, thereby making the scaling process of the frequency coefficients faster. However, this is not required, and different scaling factors can be applied to different frequency coefficients, eg, scaling higher frequency coefficients by a larger amount.

In one example, the method includes applying a scaling factor to the frequency components to determine the scaled frequency components, selecting one or more scaled frequency components according to selection criteria, and according to a one-bit encoding scheme Bit encoding of the selected scaled frequency components is performed to generate compressed image data, thereby limiting the encoding used to encode each selected scaled frequency component. However, this is not necessary and other means should be used, such as scaling after discarding some of these frequency coefficients.

In one example, the method may include selecting one of a plurality of encoding schemes and encoding the pixel data using the selected encoding scheme, which allows a different encoding scheme to be selected depending on factors such as the desired degree of compression . As such, each of these encoding schemes can provide a respective degree of compression, eg, by using different compression schemes, or using one of a plurality of different bit encoding schemes. In this latter case, each of the plurality of bit encoding schemes selectively encodes different frequency coefficients with different numbers of bits to provide different degrees of compression.

Based on a range of coefficients, such as selection rules, degree of compression desired, and/or location of pixel arrays within the one or more images, the particular encoding scheme used can be selected and then used to provide the preferred compression, There will be more detailed instructions below.

Similarly, in one example, the method typically includes selectively encoding frequency coefficients according to selection rules, desired compression levels, and/or positions of pixel arrays within the one or more images. In this regard, the selection rules can be used to define which frequency coefficients are to be encoded, and/or which particular bit-coding scheme to use, which in practice achieves the same end result and should therefore be considered an equivalent procedure.

For example: the selection rules may be used to select a subset of the frequency coefficients for encode and then encode using that bitwise encoding scheme. Alternatively, a one-bit coding scheme is selected according to a selection rule, at least some of the frequency coefficients are encoded with zero bits using the bit-coding scheme, and then the frequency coefficients are encoded according to the bit-coding scheme.

In both cases, the use of selection rules allows for selective bit encoding to be performed dynamically, so that the selected frequency coefficients and/or the number of bits used to encode the selected frequency coefficients can be adjusted according to circumstances. Similar selection rules can be used when decompressing the compressed image data, thereby allowing the bit-coding scheme and/or frequency coefficient discarding to be performed dynamically according to the particular circumstances, ensuring that the compressed data can be accurately decompressed.

In one example, this allows many different factors to be considered, such as the transmission bandwidth of a communication link used to transmit the compressed image data, the transmission quality of a communication link used to transmit the compressed image data, the display Movement of the device, image display requirements, target display resolution, an image channel processed, the location of the pixel array within the one or more images, or an observer's gaze point with respect to the one or more images the location of the pixel array within the one or more images. For another alternative, error metrics indicative of errors within the decompressed image and/or data transmission may be used in order to control the degree of compression used. It should also be understood that these configurations can be used to dynamically adjust the degree of compression, for example by changing the bit-coding scheme used to compress the image data, for example, reducing compression if the compression exceeds a critical threshold, and reducing the available transmission bandwidth if the available transmission bandwidth decreases. can improve compression. This ability to dynamically adjust compression helps optimize that compression for the best possible image quality for the current environment.

For example: the relative quality of some parts of an image may not be as important as others. In the case of virtual reality, due to the image distortion of the display lens, the peripheral part of the image is usually not actually displayed to the user. Therefore, these image parts should use a valid zero-quality encoding code, thereby greatly reducing the amount of compressed image data without losing any image quality of the viewable image.

In another example, especially in virtual reality applications, this may be performed based on an observer's gaze point. In this example, this involves determining an observer's gaze point for the one or more images, and selectively encoding frequency coefficients at least in part according to the gaze point. In particular, this may involve determining the distance between the gaze point and the position of the pixel array within the one or more images, and selectively encoding frequency coefficients according to the distance, such that the greater the distance, the fewer frequency coefficients are encoded. In this way, analysis of which part of the image the observer is looking at can be performed, for example using eye tracking techniques or similar techniques, and then the part of the image close to the gaze point is encoded with higher quality. In this regard, an observer's gaze to the surrounding area is generally reduced, so that the degradation of the image quality is generally less noticeable. Therefore, encoding images close to the viewer's gaze point with higher quality would result in an overall lower quality image being perceived by the viewer as being of average quality.

In one example, the frequency components are arranged in a plurality of layers, with each bit coding scheme defining a respective number of bits to be used to encode the frequency coefficients in each of the plurality of layers. Similarly, or with selection rules defined from which hierarchical frequency coefficients should be encoded. In this way, the selection of the bit coding scheme and/or frequency coefficients can be defined according to the respective hierarchy.

In one example, the pixel array is an NxN pixel array, producing frequency components of order 2N - 1, although it should be understood that this will depend on the particular implementation.

In one example, in addition to performing the destructive compression described above, additional non-destructive compression steps may also be performed. This usually involves parsing a sequence of bytes, identifying a subsequence containing some consistent bytes, and replacing the subsequence with values indicating those consistent bytes and some consistent bytes within the subsequence the code. In one example, when the consistent byte Where the subsequence includes three or more bytes, the code includes two bytes, although it will be appreciated that other suitable encoding schemes may be used.

While code replacement, commonly referred to as run-length encoding, can be performed on any sequence of bytes, in one example, the sequence of bytes is the bitstream formed from the encoded frequency coefficients. In this regard, there are usually many of the encoded frequency coefficients with zero values, indicating that when the bitstream formed from the encoded frequency coefficients is analyzed as a sequence of bytes, there are frequently many zero-valued bits within the sequence tuple. Thus, by replacing these with a code, this allows reducing the number of bytes.

In one example, the image data defines a plurality of channels, and wherein the method includes selectively encoding frequency coefficients for each channel. By encoding the different channels separately, this allows different encoding of the different channels, eg using different bit-coding schemes, or discarding different frequency coefficients. In addition, separate encoding channels allow channels to be encoded simultaneously, which can greatly help reduce encoding execution time, thus reducing encoding latency.

In one example, the pixel data defines RGB channels, and the method includes converting the RGB channels to luma and chrominance channels YCbCr, and transforming the YCbCr channels. In this regard, the human eye does not perceive the luma and chroma channels differently, allowing the chroma channel to be encoded with a greater degree of compression, so the quality is lower than the luma channel, but the perceived quality is lost. However, it should be understood that this is not necessary and processing could otherwise be performed within the RGB channels, in which case color conversion is not necessary.

So in this example, the method may include selectively encoding more frequency coefficients for the Y channel than the Cb or Cr channels, and similarly may include selectively encoding the Y channel with more bits than the Cb and Cr channels. Coding frequency coefficients.

In a further example, where the pixel data defines RGB channels, the method may include converting the RGB channels to YCbCr channels, and generating the compressed image data by encoding the CbCr channels and using the Y channel. In fact, in this example, the Y channel is not effectively encoded, which means that the full information contained in the luminance channel is preserved. This is particularly useful in certain encoding scenarios, such as when encoding pixel arrays that display gradients, which can help preserve color variations, thus improving picture quality, while resulting in only a small reduction in compression.

As mentioned above, these different channels can be encoded simultaneously. Furthermore, each channel of the frequency coefficients can be encoded simultaneously. In this case, the method of generating compressed image data typically includes performing concurrent byte encoding, thus serializing the frequency coefficients into a one-byte stream, followed by byte encoding.

In one example, the method of obtaining pixel data from image data includes buffering the image data corresponding to the next n -1 pixel row in the image, buffering the image data of the next n pixels in the next pixel row, and Obtain pixel data for the next n x n pixel block from the buffered image data. This process is repeated until pixel data has been obtained from all n columns of pixels, and this process is repeated for the next n columns, by buffering the image data corresponding to the next n -1 pixel columns in the image, the next pixel column buffering the image data of the next n pixels of the image data, and obtaining the pixel data of the next n x n pixel block from the buffered image data.

Thus, it can be seen from this that the process does not require the entire image to be buffered, but only that n-1 columns of pixels and a further n pixels from the next column are buffered before the process begins. This has two main impacts, in other words reducing processing time, which in turn results in a significant reduction in latency, while reducing overall memory requirements. The value of n is usually an integer and can be set based on factors such as selection rules, degree of compression desired, location of the pixel array, etc. In one example n = 8, but this is not necessary and any value can be used.

While image data can be obtained from any source, in one example, the method includes obtaining the pixel data from a video feed, such as a series of images to be displayed. In one example, the method is used to transmit virtual reality video material, and in one particular example, for wireless transmission of virtual reality video material.

The above processing can also be used to provide imagery that forms part of any digital reality content, including augmented reality, virtual reality, mixed reality, telepresence, and the like.

In one example, the method is used to display the image in a wearable digital reality headset using one of the compressed image data received from a computing device through at least one communication network and a wireless communication link material. This may include wireless transmission of compressed images from a computer to other similar devices, or may include transmission of compressed images from a cloud computing environment to a local device, such as a headset, allowing the use of cloud computing to Perform image creation. Examples of suitable connections include a wired GB Internet, streaming to a mobile phone, e.g. via a mobile communication network such as 3G, 4G or 5G network, via a wired connection to a bound HMD, or via a wireless connection Transfer to unbound HMD etc.

It should be appreciated that a similar approach may be used when decompressing the compressed image data.

For example, the same bit-coding scheme can be used, such that the bit-coding scheme uses fewer bits to encode frequency coefficients corresponding to higher frequencies. In this case, at least some frequency coefficients corresponding to higher frequencies are discarded during compression, and must be regenerated during decompression, usually as null values, allowing subsequent inverse transforms to be applied, as described in more detail below.

Typically the method involves applying a scaling factor to at least some of the frequency coefficients, thus converting the scaled frequency coefficients. Again, set the same scaling factor to Each frequency coefficient is applied, thereby increasing the strength of each frequency coefficient, thereby reversing the scaling performed during compression, and again producing an approximation of the original frequency coefficient strength.

The method may include selecting one of a plurality of decoding schemes, and decoding the pixel data using the selected decoding scheme. In this case, the decoding scheme can be selected using selection rules, the desired degree of compression, or the location of the pixel array, similar to the encoding scheme selection described above for image compression. Alternatively, this may be performed according to an encoding code indicating the encoding scheme used, the encoding code being determined by the compressed image data.

Again, the decoding scheme may be different from that described above, but the encoded frequency coefficients may likewise be decoded according to the selected bit-coding scheme using one of a plurality of different bit-coding schemes. In this case, each of the plurality of bit encoding schemes selectively encodes different frequency coefficients with different numbers of bits to provide different degrees of compression.

Similarly and effectively equivalently, the method may comprise selectively decoding the coded frequency coefficients according to a selection rule, in particular by generating the coded frequency coefficients according to a selection rule, and decoding the coded frequency coefficients according to the bit-coding scheme coefficient.

Such selection rules typically depend on a range of factors, such as one or more transmission bandwidths of a communication link used to transmit the compressed image data, a communication link used to transmit the compressed image data, such as during compression transmission quality of service, movement of the display device, image display requirements, target display resolution, a channel being processed, the location of the pixel array within the one or more images, or a The position of the pixel array within the one or more images of the viewer's gaze point.

For example, this may involve determining an observer's gaze point for the one or more images, and selectively decoding the encoded frequency coefficients based at least in part on the gaze point. This usually works This is achieved by determining the distance between the gaze point and the position of the pixel array in the one or more images, and selectively decoding the encoded frequency coefficients according to the distance, so that the greater the distance, the more frequency coefficients are generated. It should be appreciated, however, that the implementation of selection rules may be implemented in any suitable manner in accordance with the preferred implementation.

The frequency components are usually arranged in a plurality of layers, using the bit encoding scheme that defines the respective bits to be used to encode the frequency coefficients corresponding to frequency components in a respective one of the plurality of layers number, or use these selection rules, which are defined by which class of frequency coefficients should be generated.

In the event that non-destructive encoding is also performed, the method typically includes identifying a code within a sequence of bytes, and replacing the code with a subsequence containing some consistent bytes. In this case, the code typically indicates the values of the consistent bytes and some of the consistent bytes within the subsequence. Again, the subsequence typically includes three or more bytes, and the code must contain two bytes, although other suitable configurations may be used. This processing is typically performed on the compressed image data, which is used to generate the bitstream, which is then used to create the encoded frequency coefficients.

As previously explained, the image data typically defines a plurality of channels, and the encoded frequency coefficients selectively decode each channel individually. The channels typically include YCbCr channels, and the method includes performing an inverse transform of the YCbCr channels and converting the transformed YCbCr channels to RGB channels. Typically, the inverse transform is an inverse 2-D discrete cosine transform, although other suitable transforms may be used. It should also be appreciated that if the Y channel has not been encoded, as described above, the method may include decoding the CbCr channels and then converting the decoded CbCr channels and the Y channel to RGB channels.

As in the case of compressing the image data, the method typically involves generating more frequency coefficients for the Cb or Cr channels than the Y channel. The method may also include concurrently decoding the encoded The YCbCr channel also selectively bit-decodes individual frequency components simultaneously, in this case the compressed image data is at least partially decoded by serial-to-parallel byte-bit decoding, effectively distinguishing the incoming byte stream into individual bits The frequency components are meta-encoded and then decoded at the same time.

The decompressed data can also be further processed, such as using a deblocking filter to sharpen the formation of macroblocks between macroblocks when using block coding techniques, etc. Smooth edges. This in turn may allow a higher degree of compression to be used while avoiding a corresponding reduction in image quality.

In a further example, the above method is performed by a respective hardware configuration, eg, compressing image data may be performed by an encoder including an electronic encoder processing device, wherein the device obtains pixel data from the image data, performs a frequency transforming, selectively encoding at least some of the frequency coefficients using a one-bit encoding scheme, and generating compressed image data using the encoded frequency coefficients.

Similarly, decompression of the compressed video data is performed using a decoder comprising an electronic decoder processing device, wherein the device obtains the compressed video data, determines from the compressed video data a set of encoded frequency coefficients, The bit encoding scheme performs bit decoding of the encoded frequency coefficients and applies an inverse transform to the set of frequency coefficients to determine pixel data representing an array of pixels within the one or more images.

In one example, the apparatus (200) includes an encoder and a decoder in wireless communication, allowing video data to be transmitted between the encoder and the decoder as compressed video data. In one specific example, this can be used to provide wireless communication between wearable display devices, such as an HMD and a processing system. This example will now be described with reference to the second figure A.

In this example, something like a suitable programming computer system, game console, etc. A processing system 210 is adapted to generate display content on an HMD 240 . The processing system 210 typically accomplishes this by receiving sensor data from the HMD regarding the posture of the HMD, and selectively inputting data from one or more individual controllers 250 . The processing system 210 then generates content, typically in the form of video data, from the sensor and/or input data, which can be output from a video card or the like. The video data is transmitted to an encoder 220, which encodes the video data by compressing the video data, and then wirelessly transmits the compressed video data to the decoder 230 through a wireless communication link 260. The decoder 230 decodes the compressed video data and provides the resulting video data to the HMD for display.

It should be appreciated that this configuration allows existing computer systems, game consoles, etc., and HMDs 210, 240 to connect via wireless connection 260, thus avoiding the need for a wired connection between processing system 210 and HMD 240. Thus, for example, a user can wear an HMD and associated decoder, and then connect the encoder to his computer system or game console to create a wireless HMD configuration. This can be used to convert traditional tethered headsets to a wireless configuration.

This is not required, however, or the processing system 210 and HMD 240 may be configured to include integrated encoder and decoder hardware, allowing these to communicate via a direct wireless connection 260, as shown in the second panel B. For example: the encoder may be provided in a computer system for generating content, and the decoder may be integrated into a smartphone, such as a Snapdragon 820 Hexagon DSP or similar device, allowing the smartphone to receive and decode data from the Content streamed wirelessly from a computer system. In one example, this allows the computer system to stream over a local wireless connection, but in another example, this can be used to provide streaming from a cloud-based digital reality engine over a cellular network or similar content.

An example of the hardware configuration will now be described in more detail with reference to the third figure.

This example will be exemplified with separate hardware encoder and decoder, but it should be understood that this is not required and the same technique can be used in conjunction with integrated hardware. Further, while referring to virtual reality applications, again this is not required, and the technique can be applied to any environment in which image data is to be transmitted, especially when limited bandwidth is to be used to transmit image data, while maintaining acceptable images Quality and expected latency, such as in virtual reality, augmented reality or telepresence applications.

In this example, apparatus 300 again includes a processing system 310, encoder 320, decoder 330, and a display device 340 in the form of an HMD or similar. Each of these components will be described in more detail at this point.

In this example, processing system 310 includes a microprocessor 311, a memory 312, an optional input/output device 313, such as a keyboard and/or display, and a bus 315 interconnected as shown. External interface 314 . In this example, the external interface 314 may be used to connect the processing system 310 to peripheral devices, such as communication networks, storage devices, peripheral devices, and the like. Although a single external interface 314 is shown, this is only an example, and in practice multiple interfaces may be provided using many methods (eg, Ethernet, serial, USB, wireless, etc.). In this particular example, the external interface includes at least a data connection, such as USB, and a video connection, such as DisplayPort, HMDI, Thunderbolt, and the like.

In use, microprocessor 311 executes instructions in the form of application software stored in memory 312, allowing required processing to be performed. The application software may include one or more software modules and may execute within a suitable execution environment, such as an operating system environment or the like.

Thus, it should be appreciated that processing system 310 may be formed from any suitable processing system, such as a suitable programming PC or the like. In one particular example, the processing system 310 is standard processing Systems, such as Intel-based processing systems, execute software applications stored on non-volatile (eg, hard disk) storage devices, although this is not required. However, it should also be understood that the processing system may be any electronic processing device, such as a microprocessor, microchip processor, logic gate configuration, firmware optionally associated with implementing logic, such as an FPGA (field programmable gate) array), an application specific integrated circuit (ASIC), a system on a chip (SoC), a graphics processing unit (GPU), digital signal processing (DSP) or any other electronic device, system or configuration.

Furthermore, although the processing system 310 is shown as a single entity, it should be understood that in practice the processing system 310 should be formed by a plurality of physical devices, which may be selectively distributed in some geographically specific locations, such as part of a cloud environment.

The encoder 320 generally includes an encoder input buffer 321 connected to an encoder processing device 322 , an encoder output buffer 323 and a transceiver 324 in sequence. A separate data buffer 325 may be provided for coupling to transceiver 324 .

In use, image data, and in one particular example, video data has been received and temporarily stored in input buffer 321 before being sent to encoder processing device 322 for compression. In this regard, the encoder input buffer typically buffers image data corresponding to the next seven columns of pixels and then the next eight columns of pixels of the image. This allows the encoder processing device 322 to obtain pixel data for the next 8x8 block of pixels from the buffered image data and start encoding.

Once this has been done, the next eight pixels are buffered, and this step is repeated until pixel data from eight columns of pixels has been acquired and encoded. This process is then repeated for subsequent rows of pixels within the image until pixel data for the entire image has been acquired, at which point the next image is processed in a similar manner. As a result of this approach, the encoder input buffer never needs to store more than seven columns and eight pixels of image data, reducing memory requirements. In addition, for acquired pixel data material, this can be processed immediately using this encoding process, even before buffering the next eight pixels of image material. This significantly reduces processing time and helps reduce overall latency.

The resulting compressed image data is then stored in the encoder output buffer 323 , eg, by sequentially reading the encoded bits, whereby a parallel-to-serial bit is performed by the transceiver 324 before being transmitted to the decoder 330 Tuple encoding. Transceiver 324 is also adapted to transmit other data, such as a sensor data received from HMD 340, through encoder data buffer 325.

According to a preferred embodiment, the buffers 321, 323, 325 may be in the form of any suitable temporary storage device, according to the preferred implementation and in one example, may include high performance FIFO (first in first out) field memory chips, etc. Wait. The input buffer is typically connected to an HDMI port, monitor port output, or any other suitable video source, while the data buffer 335 is connected to a USB port, thereby allowing an equivalent connection to the computer system.

The transceiver 324 may be of any suitable form, but in one example, allows short-range radio communication 360 between the encoder and the decoder, such as through a point-to-point direct WiFi ^™ connection, 60GHz wireless technology, and the like.

Processing device 322 may be any device capable of performing the compression process described herein. Processing device 322 may comprise a general-purpose processing device that operates in accordance with software instructions stored in memory. However, in one example, in order to determine an appropriate fast compression time, the processing device includes customized hardware configured to perform the compression process. This should include firmware optionally associated with implementing logic, such as an FPGA (Field Programmable Gate Array), a Graphics Processing Unit (GPU), an Application-Specific Integrated Circuit (ASIC) Circuit), a system on a chip (SoC), a digital signal processor (DSP), or any other electronic device, system or configuration. in the better In an example, the encoder processing means 322 is arranged to perform parallel processing of the individual channels of each DCT, as well as parallel encoding of the individual frequency coefficients. Thus, although a single encoder processing device 322 is shown, in practice, separate encoder processing devices 322 may be provided to encode each of these channels simultaneously, or alternatively, a GPU or other similar parallel processing architecture may be used. In the event that channels such as the Y channel are not encoded, the encoder processing device can simply introduce a delay in sending the respective data to the encoder output buffer 323 to determine that this is still the same as the encoded CbCr Channel synchronization.

Decoder 330 typically includes a transceiver 334 coupled to a decoder input buffer 331 , which in turn is coupled to a decoder processing device 332 and a decoder output buffer 333 . A separate data buffer 335 is additionally provided, which is coupled to the transceiver 334 .

In use, compressed video data is received from encoder 320 via transceiver 334 and temporarily stored in input buffer 331 before being sent to decoder processing buffer 332 for decompression. Then, the resulting image data is stored in the decoder output buffer 333 before being transmitted to the display device 340 . The transceiver 324 is also adapted to transmit other data, such as a sensor data received from the display device 340 , through the decoder data buffer 335 .

According to a preferred embodiment, the buffers 331, 333, 335 may be in the form of any suitable temporary storage device and, in one example, may include high performance FIFO (first in first out) field memory chips or the like. The output buffer is usually connected to the HDMI port, while the data buffer 335 is connected to the USB port, thereby allowing an equivalent connection to the display device.

The transceiver 334 may be of any suitable form, but in one example, allows short-range radio communication 360 between the encoder and the decoder, such as through a point-to-point direct WiFi ^™ connection, 60GHz wireless technology, and the like.

The processing device 332 may include a channel that operates in accordance with software instructions stored in memory. Use a processing device. However, in one example, in order to determine an appropriately low decompression time, the processing device includes customized hardware configured to perform the decompression process. This should include firmware optionally associated with implementing logic, such as an FPGA (Field Programmable Gate Array), a Graphics Processing Unit (GPU), an Application-Specific Integrated Circuit (ASIC) Circuit), a system on a chip (SoC), a digital signal processor (DSP), or any other electronic device, system or configuration. In the preferred example, the decoder processing means 332 is arranged to perform parallel processing of the individual channels of each DCT, as well as parallel encoding of the individual frequency coefficients. Likewise, although a single decoder processing device 332 is shown, in practice, separate decoder processing devices 332 may be provided to encode each of these channels simultaneously, or alternatively, a GPU or other similar parallel processing architecture may be used. In the event that channels such as the Y channel are not encoded, the decoder processing device can simply introduce a delay in sending the respective data to the decoder output buffer 333 to determine that this is still the same as the decoded CbCr Channel synchronization.

The display device 340 includes at least one microprocessor 341, a memory 342, an optional input/output device 343, such as a keyboard or input buttons, one or more sensors 344, a display 345, and an external interface 346 , interconnected through a busbar 347 as shown.

Display device 340 may be in the form of an HMD, thus provided in a suitable housing, which can then be worn by a user, and include associated lenses to allow viewing of the display, as will be appreciated by those skilled in the art.

In this example, the external interface 347 is suitable for connecting the display device to the processing system 310 normally through a wired connection. Although a single external interface 347 is shown, this is only an example, and in practice multiple interfaces may be provided using many methods (eg, Ethernet, serial, USB, wireless, etc.). In this particular example, the external interface typically includes at least one data link connections, such as USB, and video connections, such as DisplayPort, HMDI, Thunderbolt, and more.

In use, microprocessor 341 executes instructions stored in memory 342 in the form of application software, allowing required processing to be performed. The application software may include one or more software modules and may execute within a suitable execution environment, such as an operating system environment or the like. Thus, it should be understood that the processing device may be any electronic processing device, such as a microprocessor, a microchip processor, a logic gate configuration, firmware optionally associated with implementing logic, such as an FPGA (Field Programmable Gate Array). ), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system on a chip (SoC), digital signal processing (DSP) or any other electronic device, system or configuration.

Sensors 344 are generally used to sense the orientation and/or position of display device 340 and may include inertial sensors, accelerometers, and the like. Additional sensors such as light sensors or proximity sensors can be provided to determine whether the display device is currently being worn, and eye-tracking sensors can be used to provide an indication of the user's gaze point.

In one example, the display device may thus be an existing commercial display device, such as an HTC Vive ^™ , Oculus Rift ^™ or Playstation VR ^™ headset, although it should be understood that this is not required and any suitable configuration may be used.

An example of the image compression/decompression operation will now be described in more detail.

For the purposes of this example, assume that processing system 310 is executing application software that generates content for display on display device 340, wherein based on sensor data from sensor 345 on display device 340 and optional other sensors, Content is displayed dynamically, such as a handheld controller or a position detection system (not shown), as the skilled artisan will appreciate.

Actions performed by processing system 310 are performed by processor 311 in accordance with instructions stored as application software in memory 312, and/or through I/O device 313 or other peripherals (Not shown) Receives input from the user to execute. The actions performed by the display device 340 are performed by the processor 341 according to the instructions stored in the memory 342 as application software.

The encoder 320 and the decoder 340 serve as the interface between the processing system 310 and the display device 340, allowing image data to be compressed, wirelessly transmitted and then decompressed before being displayed on the display device 340, and also allowing sensor data or input command data to be returned. to the processing system. The actions performed by the encoder 320 and decoder 330 are typically performed by the respective processing devices 322, 332 according to defined programming and, in one example, according to a customized hardware configuration and/or embedded The instructions in the firmware are executed.

It should be appreciated, however, that the above-described configuration assumed for the following example purposes is not required, and many other configurations may be used, eg, the encoder and decoder functionality may be built directly between the processing system 310 and the display device 340 Inside. In addition, the compression technique can be applied to a wide variety of other situations, including compressing and decompressing images on one or more computer systems, without the need for a separate display device. However, the above configuration is particularly advantageous for virtual or augmented reality applications, telepresence applications, and the like.

At this point, an example procedure of a method of compressing and subsequently decompressing image data will be described with reference to the fourth figures A to D. FIG.

In this example, at steps 400 and 402 , the encoder 320 receives image data, particularly video data representing a series of images, from the processing system 310 and temporarily stores it in the encoder input buffer 321 . The image data is then analyzed, eg, by parsing the data to identify flags within the data that define headers, identifying the beginning of an image, etc., in step 404 to allow the acquisition of data corresponding to the next 8x8 pixel block video material. In this regard, when buffering the data, the encoder requires an initial 8x8 pixel block from the image in order to begin processing. therefore, Before starting processing, the encoder input buffer 321 is filled with the first seven rows of pixels of an image, and the first eight pixels of the eighth row of pixels. As the next eight pixels are received, the next 8x8 block is processed, and then repeats until all pixels in the first eight columns of the image have been processed. The next eight column groups are then processed in a similar manner.

The image data is typically in the form of multi-channel RGB data, which is then converted by processing device 322 into YCbCr luminance and chrominance channels at step 406 . This processing can be performed with known mathematical coordinate transformations, so will not be described in further detail.

At step 408, a 2D DCT is applied to each of the luma and chrominance channels, thereby converting the channels to the frequency domain. This processing can be performed using known techniques and, in the preferred example, is performed by the processing device 322 with a very high degree of parallelism, thereby reducing processing time. The result of the conversion process on each channel is an 8x8 matrix with 64 frequency coefficients representing the intensities of different frequency components within the respective image channel.

At step 410, a scaling factor is applied to each matrix, eg, the scaling factor is divided by each frequency coefficient, thereby reducing the intensity of each frequency coefficient. The scaling factor should be any value and can vary from channel to channel. As part of this process, the scaled frequency coefficients are usually numerically reduced to integers or a set number of significant maps, thereby reducing the amount of data required to encode the frequency coefficients, e.g. a coefficient of 500 should require 9 bits to encode, whereas 50 requires only 6 bits, so applying a scaling factor of 10 reduces the number of bits required to encode these special frequency coefficients.

In this manner, it should be understood that lower value coefficients can be reduced numerically to zero, eg, 4 is scaled to 0.4, and thus rounded to zero. Although this results in a loss of information, it tends to occur with higher frequency components that are known to contribute less to the overall appearance of the image, so the impact of this loss is Hits are limited.

After applying the scaling factor, a selection rule is determined at step 412. The selection rules are used to allow selection of some of these frequency components so that other components may be discarded at step 414 . The selection is typically performed in terms of a hierarchy defined as a hierarchy across the matrix diagonally, such that an 8x8 frequency domain matrix includes 1, 2, 3, 4, 5, 6, 7, 8, 7, 6, respectively , 15 levels of 5, 4, 3, 2, 1 coefficients.

The layers that have been discarded are usually higher frequency components which, as mentioned above, contribute less to the appearance of the image. According to the preferred implementation, the selection rules define which tiers are discarded based on a broad range of factors.

For example, the chroma channel usually contributes less to the appearance of the image than the luma channel, so more layers of chroma are usually discarded relative to the luma channel. For example, for the luminance channel Y, levels 1 to 8 are reserved, corresponding to 36 frequency coefficients, as shown in 531 in the fifth figure, while for the chrominance channels Cb and Cr, levels 1 to 6 can be reserved, This corresponds to only 21 frequency coefficients, as shown at 532 and 533 in the fifth graph. It will be appreciated that this reduces the total number of frequency coefficients required to encode from 192 to 78 through all three channels.

Which tier to keep is selected according to the rules, and when choosing which tier to keep, the encoder and decoder are allowed to apply the same standard and then perform it in an on-the-fly adaptation manner. Then different levels are selected based on other factors, such as the quality and/or bandwidth of a communication channel between the encoder and the decoder. As such, if bandwidth is reduced due to interference, fewer layers can be selected, which results in a higher degree of compression. While this will result in lower image quality, in many applications, such as VR, some frame degradation is less noticeable than dropped frames or increased latency, and is therefore a better result.

Other factors may include, but are not limited to, movement of the display device, especially movement speed, image display requirements, target display resolution, position of the pixel array within the one or more images, and observation of the pixel array within the one or more images relative to the one or more images The position of the gaze point. Further examples are described in more detail below.

At step 416, a one-bit encoding scheme is determined, and then at step 418, a different number of bits is allowed to selectively encode the selected frequency coefficients. In this scheme, as described above, a scaled frequency component may only require 6 bits to fully encode the coefficients. Therefore, the bit encoding scheme is selected such that for each level within the hierarchy, the number of bits used to encode the frequency coefficients is changed. This works because higher frequency components generally have less intensity and therefore require fewer bits to encode.

In one example, the encoding scheme used for the luma channel is different from that used for the chroma channel, and example encoding schemes are shown in Table 1 below.

An example of this form of encoding for the first 6 levels in each of the three channels is shown in the fifth figure. Using this particular combination of selection rules and bit encoding scheme results in encoding the luma channel 541 with 129 bits, and encoding each of the chroma channels 542, 543 with 60 bits, resulting in the 8x8 being encoded using only 249 bits All three channels of the pixel array. It should be understood that this is compared to Requires 192 bytes of raw uncompressed image data, representing more than six times the compression.

It will be appreciated from this that an encrypted channel with 0 bits effectively corresponds to discarding the channel, so this can be used as a way of selecting the encoded coefficients by selecting an encoding scheme that uses the encoding rules, thereby effectively Combine steps 412 and 414 with 416 and 418.

Once the encoding has been performed, by performing parallel-to-sequence byte encoding at step 420, the bit-encoded frequency components can be concatenated into a one-bit stream, which then allows 32-byte (256-bit) )express. At step 422, the bytes are parsed to identify subsequent consistent bytes at step 424. In particular, this approach is used to identify subsequences of three or more identical bytes, which are then replaced with a code at step 426 without any loss of information.

Especially for most images, there is a string of zeros in the resulting encoded frequency coefficients, where the scaled coefficients are reduced to zero in value. Therefore, it can be replaced with a code that can be recognized by the decoder, allowing the decoder to re-insert the subsequence of consistent bytes.

Although the code may be in any suitable form, in one example, the code includes a header identifying the particular byte as the code, and information corresponding to the value and number of the corresponding byte. In the preferred configuration, a Boolean OR operation is used to combine a 2-byte code with a zero number in a column (1-8). In one example, the zero number represents N-1, so the numbers 0-7 are ORed with the 2-byte code, so that only 3 bits of the second byte are used, for example: the used The code may be (1111 1111; 1111 1000), where the second byte is ORed with 0-7 according to the zero number. It should be appreciated that a similar approach can be used for different values.

This works well because the encoding rarely results in consecutive numbers exceeding or equal to the value of 248, so the decoding algorithm can simply search for a byte with a value of 255 and subsequent bytes with a value greater than or equal to 248, converting It is identified as the inverse of the coded frequency components code. This code is then replaced with the byte corresponding to the data having a series of zeros represented by the last 3 bits in the second bit. This can result in a further 19-25% reduction in data after this bit encoding stage according to the latest tests.

After performing the code replacement, at step 428 the compressed image data is output. In particular, the compressed image data is usually stored in the output buffer 323 until there is enough data, at which point a data packet is created and transmitted by the transceiver 324 to the encoder.

At step 430 , the compressed data is received by the decoder 330 through the transceiver 334 and stored in the decoder input buffer 331 . The data is parsed at step 432 to identify codes within the data, as described above, and then replaced at step 434 with a repeating subsequence of consistent bytes, thereby recreating the bits of the bit-encoded frequency coefficients stream.

At step 436, decoder processing device 332 determines the bit encoding scheme used to encode the image data, and uses this scheme to perform sequential-to-parallel decoding of the bitstream, thereby determining at step 438 the already Coding frequency coefficients. In particular, this allows decoder processing means 332 to determine the number of bits used to encode the frequency coefficients, allowing the bitstream to be efficiently divided into individual encoded frequency coefficients. The individual frequency coefficients are used to recreate the scaled frequency matrix at step 440 , the discarded frequency coefficients are generated using nulls, thereby filling the entire matrix, and then at step 442 is scaled according to the scale factor.

At step 444, an inverse 2D DCT transformation is applied prior to converting the converted matrix of each YCbCr channel to RGB channels at step 446, allowing an 8x8 pixel block to be output at step 448, allowing implementation by display device 340 .

Therefore, the above process can significantly reduce the amount of image data required to encode each 8x8 pixel block, thus reducing the overall image size. In particular, 2D DCT using YCbCr channels, for Selective bit-encoding of the resulting frequency coefficients and an optional final non-destructive encoding scheme accomplishes this. Each channel can be processed simultaneously, applying the DCT to the 8x8 array in parallel. Furthermore, the bit-coding of each of these frequency coefficients also occurs simultaneously, resulting in a large number of parallel methods, allowing compression and decompression to be performed simultaneously in a rapid manner with minimal impact on latency, which is the case in applications such as VR, AR, and remote. It is very important in this kind of instant application.

An example of the manner in which the selection rule is implemented will now be described with reference to the sixth and seventh figures.

In particular, this example focuses on how the spatial positioning of the pixel array within an image is used to influence the selection process, and thus the amount of compression performed. Similar processing may be performed for other factors and will not be described in detail herein. This process is the same for either the encoder or the decoder, so for ease of illustration, only the operation of the encoder will be focused.

In this example, at step 600, the encoder processing device 322 determines the selection rule for the current channel. In this regard, as previously described, different rules are typically applied to each of the luma and chroma channels. Then at step 605, the encoder processing device 322 determines the position of the pixel array within the image, and from there, determines whether an absolute rule is associated with the position. In this regard, when a headset is used to display images, the lens configuration typically means that the portion of the image is not visible. This example is shown in Figure 7, which shows a viewable portion 701 and an unviewable portion 702 of an image. In this regard, if the pixel array is within the non-viewable portion 702, the pixel array does not need to be encoded, so the process can simply ignore all levels of the frequency coefficients. Therefore, if there is a rule for the absolute position of the pixel array at step 610, then the process proceeds to step 625, allowing the individual level selections to be determined, and then at step 630 using the respective bit encoding scheme, as above with respect to steps 414 to 418 instruction of.

Otherwise, at step 615, the encoder processing device 322 determines the viewer's gaze point for the image. This can be achieved using an eye-tracking sensor system that is accessible to the tech-savvy. Once determined, a relative position is determined at step 620, and the distance between the gaze point and the current pixel array position within an example is determined, which is then used at steps 625 and 630 to guide coefficient and/or bit encoding program decision. In this regard, when one's peripheral field of view does not perceive the same level of detail as its focal position, a greater degree of compression can be used further from the user's gaze point.

In one example, a region of lowest compression may be defined around the user's gaze point, and the degree of compression increases as one moves outward from the lowest region. The region can be of any shape and can be circular, oval, oval, etc. depending on the particular circumstances. This area can also be off-center relative to the gaze zone, so make sure to use the minimum compression below the gaze point, which is generally the area that the user is more perceptible to.

It should also be appreciated that in environments where the user's gaze point cannot be measured, this is predictable. In general, the prediction is focused on the center of the frame, but it should be understood that this is not required and may vary, eg, depending on things like the nature of the content represented, the direction in which the headset is moving, and so on.

As such, it should be appreciated that this illustrates a mechanism that allows selective encoding to be performed based on the absolute and/or relative position of the pixel array within the image. Having the encoder and decoder each perform the same algorithm to select the encoding scheme and the selection used allows the decoder to reliably and accurately decompress the compressed image data, while still allowing encoding to be performed dynamically, The overall compression is maximized while minimizing the perceived loss of image quality.

It should be appreciated that a similar approach can be used for other factors, such as transmission quality and/or bandwidth, and thus determine the level of compression best suited to the current environment.

A further example of a method for compressing image data and subsequently decompressing the compressed image data will now be described with reference to FIG. 8 .

Although illustrated with a separate example, it will be appreciated from the following description that the current example can be used in conjunction with the above-described compression/decompression techniques to further improve compression and computational performance. In particular, the above-described technique represents one of the options that can be employed in the following processing, although it will be appreciated that other compression techniques may be appropriate depending on the circumstances.

In this example, at step 800, pixel data representing an array of pixels within the one or more images is obtained from pixel data. This can be accomplished in any suitable manner, but typically involves buffering the received image data until an array of pixels, such as an 8x8 array, has been obtained in a manner similar to that described above.

At step 810, an encoding scheme has been determined, typically selected from one or more previously defined encoding schemes, such as the encoding schemes described above with respect to Figures 1-7. The selection may be performed in any suitable manner, such as through analysis of the image and/or pixel data, based on the type of image associated with the image, based on instructions from an image source or display, etc., and is generally based on the specifics of the current situation requirements, implemented to provide a higher or lower degree of compression.

At step 820, the pixel data is encoded using the encoding scheme, eg, using the techniques described above and the like. At step 830, an encoding code indicating the encoding scheme is used, for example, through a look-up associated with the available schemes, which are used in conjunction with the encoded block to generate compressed image data. In one example, this may be accomplished using the encoded code as a prefix code added to the encoded pixel data, although other means may be used, including simply by replacing the pixel with the encoded code, depending on the encoding scheme used Data, for example, is replaced with a code character.

Once the compressed image data has been created, this can then be transmitted to a destination, such as part of a one-byte stream, allowing decoding to generate the image data.

At step 840, this process is typically performed by determining the encoding code from the compressed image data. In this regard, the encoded code typically has a set of formats, such as a specific combination of bits, that can be easily identified within the received stream of bytes. The encoding code is then used to determine the encoding scheme to use, typically through the lookup procedure at step 850 . Then at step 860, the compressed image data is decoded using the identified encoding scheme, such as in the manner described above with respect to Figures 1-7, thereby generating a representation of the image within the one or more images. Pixel data for the pixel array.

Thus, the above process allows encoding to be performed using one of several encoding schemes. In the preferred example, the encoder and decoder are known in advance to use these different encoding schemes, allowing dynamic changes for each pixel array within the image as encoded/decoded. This determines that the encoding scheme is used to optimize the particular data being encoded, thereby maximizing compression, while determining to maintain other desired characteristics, such as image quality and encoding time.

Some further features will be described at this point.

When encoding the image, the processing typically involves one or more of a range of factors, including the image type of the image data, an encoding scheme indication received from an image data source or display, analysis of at least the image using One of the data and the pixel data, or according to compression requirements, such as the amount of compression required, resulting image quality, compression delay, etc., determines the encoding scheme.

Thus, for example, when encoding photographs rather than computer-generated graphics, different encoding types can be used, as they often have very different properties. From the metadata associated with the image, Or according to the properties of the image itself, the properties of the image can be obtained. Additionally, an image source that supplies the image data, such as a computer system that supplies the image data for transmission to a display or other computing device, can specify the encoding scheme that should be used.

A further example is to analyze the pixel data to identify the characteristics of the pixel array, eg if the pixel array has a single color, a different algorithm should be used than that used for the pixel array containing multiple colors. This is especially useful for encoding large area images with a single color, such as the sky or background, allowing a single encoding code that indicates the use of a solid color to replace the pixel data for the entire pixel array, thereby resulting in maximum compression with no effect on image quality loss.

A further example is the identification of gradients or boundaries, which may result in the use of many image compression methods still not obtaining the desired compression, and thus an alternative compression method may be required.

In a preferred example, where the pixel array is not a solid color, the approach described with respect to Figures 1 to 7 is used. In the event that the pixel array contains no gradients or boundaries, this technique may involve encoding all three of the YCbCr channels. However, in the event that the pixel array contains a gradient or boundary, the Y channel may not be encoded, thus preserving additional luminance information. Although this reduces the final amount of compression obtained, the amount of compression is still sufficient due to the compression of the CbCr channels, and the additionally preserved color information can significantly improve the quality of the resulting image in the gradient or border region.

This selection can also be used to control other aspects of a compression algorithm, eg, this applies to specifying which frequency coefficients are to be preserved in the compression process described above in a manner similar to that performed with respect to the process in the sixth figure.

The encoding code is usually specified as a number from 0 to 255, which can be defined with a single tuple, and allows up to 256 different encoding schemes to choose from. so widely varied Coding schemes can be used, and these schemes are assigned to numbers 0 to 255, respectively, so that a wide variety of coding schemes can be used, thereby achieving the best compression with the desired image quality. In one particular example, it should be understood that an industry standard list of encoding schemes can be established so that the encoding and decoding systems can work together even if the encoding scheme to be used has not been previously communicated.

In one specific example, the encoding code can be identified by a fixed format leading byte such as "11111111", so the encoding code used to select the encoding scheme assigned to the number 1 should be "1111111100000001". It will be appreciated that doing this allows the decoding process to easily identify the encoding process used. However, any suitable approach may be used and the above examples are not intended to be limiting.

It will be appreciated that a similar approach is used when decoding the compressed image data, including replacing an encoded code with a solid-color pixel array, or using the decoding process previously described.

It should also be appreciated that the image compression process may be performed using equipment similar to those described above, and thus will not be described in further detail.

Referring now to Figure 9 A and Figure 9 B, further example processing is described in more detail.

In this example, at step 904 , the encoder 320 receives image data, particularly video data representing a series of images, from the processing system 310 and then temporarily stores it in the encoder input buffer 321 at step 902 . The image data is then analyzed, eg, by parsing the data to identify flags within the data defining the header, identifying the beginning of an image, etc., and obtaining image data corresponding to the next 8x8 pixel block at step 904 . It should be appreciated that this corresponds to steps 400 to 404 described above, and thus will not be described in further detail.

At step 906, the encoder 920 selects an encoding scheme to use. It's transparent Through the analysis of the 8x8 pixel block, for example, it is determined whether the block is a single color, gradient, edge, etc., or can be realized according to the information provided by the image source.

In the example shown, the encoding schemes include a first encoding scheme for performing a block replacement at step 908 in which a block of monochrome pixels is replaced with a known predetermined code. A second encoding scheme is provided at step 908, which in this example corresponds to the encoding scheme described above, and assigned to steps 406-426. However, it should be appreciated that any number N of different schemes can be used, and then other schemes are selected and used at step 910.

In step 912, an encoding code representing the selected encoding scheme is prepended to the encoded data, and then in step 914 the data is output as compressed image data.

Next, in step 916, the decoder 330 receives the compressed image data to perform the decoding, and in step 918, detects the presence or absence of the encoding code, and then, in step 920, uses the detection result to select a proper decoding scheme. For example: this may include performing block replacement at step 922, replacing the predetermined code with a block of monochrome pixels, or performing the decoding process of steps 432-446 at step 924. Again, N suitable decoding schemes are provided, while other schemes may be selected and used at step 926 as desired.

Finally, at step 928, an 8x8 pixel block is output from the decoder.

From this, it can be understood that the selection process performed at step 906 performs the classification function to be performed, so that each pixel block can be assigned to a different encoding scheme. In one example, up to 256 different schemes can be defined, allowing third parties to import new compression/decompression schemes to the 8x8 pixel block within the previously described architecture. Furthermore, the user can define a specific classification scheme, such as according to the nature of the image content to be encoded, the requirements of the situation, such as delay requirements, etc., For example, different combinations of 256 schemes can be used in different environments.

This greatly increases the flexibility of the encoding that can be performed, and in particular allows the use of a per-pixel block classification process to dynamically select individual encoding schemes, thereby optimizing the encoding of each block on a case-by-case basis. Still further, different sorting schemes can be provided, each different sorting scheme allowing access to different combinations of coding schemes, thereby further increasing flexibility, allowing the user to ensure that optimal coding can be performed in a variety of different situations.

In another broad form, the motion compression scheme described in Figures 6 and 7 above can also be used in conjunction with different encryption schemes to provide a way to combine images from one or more images forming part of digital reality content. The method of image data compression will now be described with reference to Figure 10.

In this example, the method includes obtaining pixel data from the image data at step 1000, the pixel data representing an array of pixels within the one or more images. This can be accomplished in a manner similar to that described above.

At step 1010, a location of the pixel array within the one or more images is determined relative to a defined location that at least partially indicates a gaze point of the user. The defined position can be based on the actual measured point of gaze, or the user's expected or predicted gaze point, such as by assuming the user is starting to stare at the approximate center of the image, or based on content, such as focus within the image, head-mounted The movement of the group etc. is decided. Additionally, the defining point may be offset from the gaze point, eg, positioned below the gaze point, to account for a personal tendency to focus slightly below the gaze point to avoid bumping into obstacles while walking.

Next, in step 1020, the pixel data is compressed to generate compressed image data, and the pixel data is compressed at least in part according to the determined position, such that the degree of compression is Depends on the determined position of the pixel array.

It will be appreciated that a similar decompression process may be performed involving obtaining compressed image data at step 1030, the compressed image data representing an array of pixels within the one or more images, and based at least in part on the The array of pixels within one or more images is compressed relative to a position at a defined location that at least partially indicates the user's gaze point, and at step 1040 is at least partially based on the determined location Decompress the compressed image data.

As such, this provides a mechanism for compressing and subsequently decompressing the image, wherein the compression is controlled according to the position of an array of pixels relative to a defined point, and then at least in part according to the user's predicted or measured gaze point. In particular, this allows the degree of compression to be chosen based on the location of the pixel array, so that less compression can be used in areas near the gaze point, and more compression can be used in areas far from the gaze point, such as within the periphery of the user's field of view. This actually provides foveated compression, by increasing the compression of the user's peripheral field of view, where the reduced image quality is less noticeable, allowing for greater overall compression with no perceptible loss of quality.

It will be appreciated that this process can be used in conjunction with the compression process described above, eg, to allow selection of different compression schemes based on the location of the pixel array, and to allow the use of the bit-coding schemes described above. However, this is not required and any suitable compression scheme can be used, like wavelet compression, adaptive sampling, etc.

In any event, some further features will be described at this time.

In one example, the defined locations are the user's gaze measurement point, the user's gaze expected point, the deviation from the user's gaze measurement point, the deviation from the user's expected gaze point, and at least in part in accordance with the point of indicating the user's gaze One of the points determined by the gaze data, where the gaze Visual data is obtained from a gaze tracking system. As such, the defined point may be based on a measured or predicted point of gaze and may be offset, such as below the gaze point, in order to optimize the degree of compression without causing an appreciable reduction in image quality.

The degree of compression may be based on the distance from the definition point, eg, gradually decreases away from the gaze point, but may also be based on the direction relative to the definition point, so that the compression is more above or below the definition point. It will be appreciated that this allows a corresponding degree of compression to be used in locating regions of arbitrary shape relative to the gaze point, and this may be configured according to the specific environment and/or properties of the compressed content. For example, this allows an elliptical, oval or heart-shaped area around the defining point to have a reduced amount of compression compared to the surrounding area, thereby optimizing image quality in areas where the user's perception of any compression will be greater .

In one example, the method includes selecting one of a plurality of encoding schemes, and encoding the pixel data using the selected encoding scheme, as will be appreciated, this allows incorporating techniques similar to those described in Figures 8 and 9 to perform rendering compression, where the encoding scheme used is chosen according to the location of the pixel array. Thus, in one example, each encoding scheme provides a corresponding degree of compression, and wherein the method includes selecting an encoding scheme based, at least in part, on a desired degree of compression and/or location of the pixel array.

The process also involves determining an encoding code indicating the encoding scheme used, and using the encoding code and encoded pixel data to generate compressed image data, allowing it to be used to decode the compressed image data. However, this is not required, or the decompression process may involve the use of similar criteria to determine the encoding scheme used, such as the location of the pixel array and thus the degree of compression required.

In one example, the encoding scheme uses a coding scheme similar to that described above with respect to the first figure In the manner described, in this case, the image data is compressed by applying a transformation to the pixel data to determine a set of frequency coefficients indicating the frequency components of the pixel array, selectively encoding at least some of these using a one-bit encoding scheme frequency coefficients, thereby generating a set of coded frequency coefficients, and using the coded frequency coefficients to generate compressed image data. It should be appreciated, however, that any or more suitable compression techniques may be used, such as JPEG or the like.

In the preferred example, as in the case of the first figure above, the bit encoding scheme defines the number of bits used to encode each of the frequency coefficients, such that at least some of the encoded frequency coefficients have different bits arity. More typically, a smaller number of bits are used to encode frequency coefficients corresponding to higher frequencies, and preferably progressively smaller numbers of bits are used to encode frequency coefficients corresponding to progressively higher frequencies. In one example, at least some of the frequency coefficients are discarded (effectively encoded with zero bits) so that the set of coded frequency coefficients is smaller than the set of frequency coefficients, and which generally correspond to higher frequencies.

In one example, the method includes selecting one of a plurality of bit encoding schemes, and encoding the frequency coefficients according to the selected bit encoding scheme. It should thus be appreciated that different encoding schemes may correspond to the encoding scheme of the first figure, wherein different bit encoding schemes are used to encode different frequency coefficients with respective different numbers of bits, thereby providing different degrees of compression .

As will be appreciated, the bit encoding scheme may be selected based at least in part on the desired degree of compression and/or the location of the pixel array. This may depend on other factors such as the location of the pixel array, the transmission bandwidth of a communication link used to send the compressed image data, the transmission quality of a communication link used to send the compressed image data, a display device movement, image display requirements, a target display resolution, the channel being processed, the error matrix, etc. As above, This allows the ability to dynamically adjust the compression to help optimize the compression and achieve the best possible image quality under the current circumstances.

In one example, the frequency components are arranged in a plurality of levels, and wherein each bit encoding scheme defines an additional number of bits to be used to encode the frequency coefficients in each of the plurality of levels. In particular, where the array is an NxN array of pixels, this produces frequency components of order 2N - 1, wherein the components within each level are encoded with a respective number of bits.

As described above, the method may include applying a scaling factor in a manner similar to that described above.

The image data typically defines a plurality of channels, wherein the method includes selectively encoding frequency coefficients for each channel. For example, the pixel data can define RGB channels, these channels are converted to YCbCr channels, and then the channels are converted as described above. This process involves selectively encoding more frequency coefficients to the Y channel than the Cb or Cr channels, selectively encoding frequency coefficients simultaneously to the YCbCr channels, and/or selectively encoding frequency coefficients to the CbCr channels and using the Y channel, so the Y channel is effectively uncompressed.

In one example, the transform is a 2-D discrete cosine transform.

The method also typically includes buffering the image data to obtain the array of pixels, eg, by buffering n-1 columns of pixels, and the next n pixels of the next column of pixels, to form a first n x n block of pixels. Consecutive blocks are then buffered as previously described, wherein the number of pixels in each block can be selectively controlled based on the selected bit encoding scheme, the desired degree of compression, and/or the location of the pixel array.

The frequency coefficients are usually encoded simultaneously and the compressed image data is generated by parallel-to-sequential byte encoding, although other suitable methods may be used.

A similar approach can be used to decode the compressed video data.

For example, decoding the compressed image data may include selecting one of a plurality of decoding schemes, and decoding the pixel data using the selected decoding scheme, wherein the selection is based on a desired degree of compression, the location of the pixel array, or the An encoding code within the compressed image data.

The method may include using a decompression scheme similar to that described above, which may provide varying degrees of compression using one of a plurality of bit encoding schemes, by encoding individual bits for the frequency coefficients in each of the plurality of levels arity.

A scaling factor may be applied to at least some of the frequency coefficients, such that the scaled encoded frequency coefficients are decoded, the scaling factor being used to increase the strength of each frequency coefficient. Again, different scaling factors may be applied to at least some of the encoded frequency coefficients, or the same scaling factor may be applied to each encoded frequency coefficient. In both cases, different scaling factors can be applied to the encoded frequency coefficients in different channels.

A plurality of channels may be used, with each channel being decoded as needed, which may be selectively performed simultaneously by decoding the frequency coefficients simultaneously using sequence-to-parallel bit-set decoding and selectively decoding the frequency coefficients simultaneously.

The degree of compression desired by the above techniques can be based on the location of the pixel array, the transmission bandwidth of a communication link used to send the compressed image data, the transmission quality of a communication link used to send the compressed image data, a Any one or more of movement of the display device, image display requirements, a target display resolution, the channel being processed, or an error matrix.

These techniques may be used to transmit digital reality content, including any or more of augmented reality, virtual reality, mixed reality, telepresence, and the like. This may include sending the image data from the computing device to the wearable digital reality headset via a communication network and/or a wireless communication link.

Thus, the above processing allows for significant image compression without adversely affecting latency, allowing the technology to be used to provide wireless connectivity to virtual or augmented reality wearable displays using existing hardware, while also allowing for other Improved image compression for wider application.

In this specification and the scope of the following claims, unless the context requires otherwise, the word "comprising" and variations such as "comprising" or "comprising" will be understood to imply inclusion of the stated group of integers or steps, but Any other integers or groups of integers are not excluded.

Tech-savvy people will understand that many changes and modifications will become apparent. All such changes and modifications that should be apparent to those skilled in the art are intended to fall within the broad spirit and scope of the invention described above.

100-170‧‧‧Steps

Claims (20)

A method of compressing image data representing one or more images, the method comprising: a) obtaining pixel data from the image data, the pixel data representing an array of pixels within the one or more images; b) pairing Apply a transformation to the pixel data to determine a set of frequency coefficients indicative of the frequency components of the pixel array; c) select one of a plurality of bit-coding schemes, wherein each of the bit-coding schemes uses a different bit Binary selective encoding of different frequency coefficients to provide different degrees of compression, wherein the bit encoding scheme is selected based at least in part on: i) a desired degree of compression; and ii) the one or more images a position within the pixel array; and d) selectively encoding at least some of the frequency coefficients using the selected bit encoding scheme, thereby generating a set of encoded frequency coefficients, wherein the bit encoding scheme defines a in encoding the number of bits of the frequency coefficients such that when the frequency coefficients are selectively encoded: i) at least some of the encoded frequency coefficients are encoded with a different number of bits; and ii) at least one frequency coefficient is discarded so that the set of coded frequency coefficients is smaller than the set of frequency coefficients; and e) using the coded frequency coefficient to generate compressed image data. The method of claim 1, wherein a bit encoding scheme is selected based on at least one of the following: a) a wireless communication link used to transmit the compressed image data to a display device transmission bandwidth; b) transmission quality of a wireless communication link used to transmit the compressed image data to a display device; c) lens arrangement of a display device; d) movement of a display device; e) a display device image display requirements; f) the target display resolution of a display device. The method of claim 1, wherein the frequency coefficients are selectively encoded such that at least one of the following: a) defines the use of a number of bits to encode each of the frequency coefficients, and the value of the respective frequency coefficient irrelevant; b) fewer bits are used to encode frequency coefficients corresponding to higher frequencies; c) progressively fewer bits are used to encode frequency coefficients corresponding to progressively higher frequencies; d) discarded at least one frequency coefficient for higher frequencies. The method of claim 1, wherein the method comprises applying a scaling factor to at least some of the frequency coefficients, thus encoding the scaled frequency coefficients, and wherein at least one of the following: a) applying a different scale A scaling factor is applied to at least some of the frequency coefficients; b) the same scaling factor is applied to each frequency coefficient; and c) the scaling factor is used to reduce the intensity of each of the frequency coefficients. The method of claim 1, wherein the frequency components are arranged in a plurality of levels, and wherein each bit coding scheme defines a value to be used to encode the frequency coefficients in each of the plurality of levels A respective number of bits. The method of claim 1, wherein the method comprises: a) determining a gaze point of an observer of the one or more images; b) selectively encoding frequency coefficients at least in part according to the gaze point: i) determining the distance between the gaze point and a position of the pixel array within the one or more images; and ii) selectively encoding frequency coefficients according to the distance, such that larger distances have fewer frequency coefficients encoded. The method of claim 1, wherein the image data defines a plurality of channels, and wherein the method includes selectively encoding frequency coefficients for each channel by: a) converting the RGB channels to YCbCr channels in parallel; and b) Converting the YCbCr channels by selectively encoding more frequency coefficients for the Y channel than the Cb or Cr channels. The method of claim 1, wherein the method comprises obtaining pixel data from image data using: a) buffering image data corresponding to the next n -1 rows of pixels in the image; b) buffering the image data of the next row of pixels image data for the next n pixels; c) obtain pixel data for the next block of n x n pixels from the buffered image data; d) repeat steps b) and c) until all the n rows of pixels have been obtained pixel data; and e) repeating steps a) to d) until pixel data has been obtained from each column of pixels of the image. The method of claim 1, wherein the method comprises: a) simultaneously selecting encoding frequency coefficients; and b) generating compressed image data at least in part by parallel-to-sequential byte encoding. A decompression method for compressed image data, the compressed image data representing one or more images, the method comprising: a) obtaining the compressed image data; b) determining a selected one of a plurality of bit encoding schemes, wherein each of the bit-coding schemes selectively encodes different frequency coefficients at respective different numbers of bits to provide different degrees of compression, wherein the bit-coding scheme is selected based on at least i) a desired degree of compression; and ii) a position of the pixel array within the one or more images; c) decoding the encoded frequency coefficients according to the selected bit encoding scheme by: i) according to the selected A bit-coding scheme determines a set of coded frequency coefficients from the compressed image data; ii) performs bit-wise decoding of the frequency coefficients according to the selected bit-coding scheme to generate a set of frequency coefficients, wherein at least one frequency coefficients are generated such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and iii) applying an inverse transformation to the set of frequency coefficients to determine pixel data representing an array of pixels within the one or more images. The method of claim 10, wherein the bit encoding scheme is selected based on at least one of: a) the transmission bandwidth of a wireless communication link used to transmit the compressed image data to a display device; b) the transmission service quality of a wireless communication link used to transmit the compressed image data to a display device; c) the lens arrangement of a display device; d) movement of a display device; e) image display requirements of a display device; f) target display resolution of a display device. The method of claim 1 or 10 of the scope of the patent application, wherein the digital reality is at least one of the following: a) augmented reality; b) virtual reality; and c) mixed reality. The method of claim 1 or 10, wherein the method is for displaying image data in a wearable digital reality headset by receiving the compressed image data from a computer device via at least one of the following: a ) a communication network; and b) a wireless communication link. The method of claim 1 or 10 of the claimed scope, wherein the method is used for at least one of the following: a) transmitting virtual reality video data; and b) wirelessly transmitting virtual reality video data. An apparatus for compressing image data representing one or more images, the apparatus comprising at least one electronic encoder processing means, the at least one electronic encoder processing means: a) obtaining pixel data from the image data, the pixel data representing a pixel array within the one or more images; b) applying a transformation to the pixel data to determine a set of frequency coefficients indicative of frequency components of the pixel array; c) selecting one of a plurality of bit encoding schemes , wherein each of the equal bit encoding schemes selectively encodes different frequency coefficients with respective different numbers of bits to provide different , wherein the bit-coding scheme is selected based at least on: i) a desired degree of compression; and ii) a position of the pixel array within the one or more images; d) using the selected A bit-coding scheme selectively encodes at least some of the frequency coefficients, thereby producing a set of coded frequency coefficients, wherein the bit-coding scheme defines the number of bits used to encode the frequency coefficients, and wherein the When frequency coefficients are selectively encoded: i) at least some of the encoded frequency coefficients have different numbers of bits; ii) discard at least one frequency coefficient, so that the group of encoded frequency coefficients is smaller than the group of frequency coefficients; and e) use the already encoded frequency coefficients The encoded frequency coefficients produce compressed image data. The apparatus of claim 15, wherein the at least one electronic encoder processing device selects the bit encoding scheme based on at least one of the following: a) a wireless communication link for transmitting the compressed image data to a display device transmission bandwidth; b) transmission quality of a wireless communication link used to transmit the compressed image data to a display device; c) lens arrangement of a display device; d) movement of a display device; e) a display The image display requirements of the device; f) the target display resolution of a display device. The apparatus of claim 15 or 16 of the claimed scope, wherein the apparatus comprises an encoder communicatively connected to a decoder to allow image data to be converted between the encoder and the decoder as compressed image data, wherein the The encoder is a processing system coupled to a suitable programming at least one and a part thereof, the decoder is coupled to at least one and a part thereof of a wearable display device. The apparatus of claim 15, wherein the at least one encoder processing device comprises: a) a suitably programmed field programmable gate array; b) a dedicated integrated circuit; and c) a graphics processing unit. 15. The apparatus of claim 15, wherein the pixel data defines a plurality of channels, and wherein the apparatus includes: a) a separate processing device for each channel; and b) a simultaneous processing device for each channel Parallel processing device. An apparatus for decompressing compressed image data representing one or more images, the apparatus comprising at least one electronic decoder processing means, the at least one electronic decoder processing means: a) obtaining the compressed image data; b) determining a selected one of a plurality of bit-coding schemes, wherein each of the bit-coding schemes selectively encodes different frequency coefficients at respective different numbers of bits to provide different degrees of compression, wherein the bit-coding schemes The scheme is selected based at least on: i) a desired degree of compression; and ii) a position of the pixel array within the one or more images; c) decoding the bit-coding scheme according to the selected by The encoded frequency coefficients: i) determine a set of encoded frequency coefficients from the compressed image data by following the selected bit encoding scheme that defines the number of bits used within each encoded frequency coefficient; ii) executing the encoded frequency coefficients according to the selected bit-coding scheme decoding of the bits of Pixel data for a pixel array within one or more images.

Images