TW201806386A - Image compression method and apparatus - Google Patents

Abstract

A method of compressing image data from one or more images forming part of digital reality content, the method including obtaining pixel data from the image data, the pixel data representing an array of pixels within the one or more images; determining a position of the array of pixels within the one or more images relative to a defined position, the defined position being at least partially indicative of a point of gaze of the user; and compressing the pixel data at least partially in accordance the determined position so that a degree of compression depends on the determined position of the array of pixels.

Description

Image compression method and device

The present invention relates to a method and apparatus for compressing or decompressing image data, and in one particular example for compressing or decompressing image data, allowing image data to be transmitted with reduced bandwidth and less delay.

References in this specification to any prior disclosure (or derivative information) or any known matter are not and should not be considered as confirmation or recognition or any form of recommendation, ie prior application (or derivative information) or known matter. The object constitutes part of the general knowledge of the business areas covered by this specification.

In virtual, enhanced and hybrid real-world systems, a wearable display device, such as a Head Mounted Display (HMD), is generally provided that displays information based on the relative spatial position and/or orientation of the display device. Wearer. Such a system operates by generating images based on information about the position (position and orientation) of the display device such that as the display device moves, the images are updated to reflect the new pose of the display device.

In order to avoid dizziness, the time difference between the collection of posture information and the establishment of the corresponding image is minimized, especially in the case of rapid movement of the display device. Combining the production of high-resolution images makes them look as realistic as possible, which means that there is a need for fairly high-level processing hardware. As a result, high-end existing systems typically require a static desktop computer that has a high frequency bandwidth and low latency to connect to the display device. Thus, like the HTC Vive ^TM, Oculus Rift ^TM Playstation VR ^TM and these current systems require a wired connection between the computer and of the HMD, which is not convenient.

Although there are actions available solutions, such as Gear VR ^TM, which is incorporated into the mobile phone to perform image processing and display in the HMD itself, but the processing capacity is limited, meaning content can be displayed is limited, especially in image resolution And quality aspects.

We all know that compressing image data can reduce the volume of data. This is useful in many applications, such as reducing the storage capacity required to store the image data, or reducing the bandwidth requirements associated with transmitting the image data.

JPEG uses destructive compression according to discrete cosine transform (DCT), which converts each frame/field of a video source from a spatial (2D) domain to a frequency domain (ie, a transform domain). ). The loosely based perception model of the human mental vision system discards high-frequency information, that is, sharp changes in intensity and hue. In the conversion domain, information is reduced by quantification. The quantized coefficients are then sequenced and loosely packed into an output bit stream.

However, this approach typically only achieves a limited amount of compression and requires considerable processing time and is therefore not suitable for low latency applications such as virtual or augmented reality, telepresence, and the like.

In a broad form, aspects of the present invention seek to provide a method of compressing image data in one or more images forming part of a digital reality content, the method comprising: obtaining pixel data from the image data, the pixel The material data represents a pixel array within the one or more images; determining the pixel array within the one or more images relative to a defined position a location, the defined location at least partially indicating a gaze point of the user; and compressing the pixel data to generate compressed image data, the pixel data being compressed at least in part in accordance with the determined position, such that the degree of compression depends The determined position of the pixel array.

In a specific embodiment, the defined location is at least one of: a measured user gaze point; an expected user gaze point; and an offset of the measured user gaze point; Offset to an expected point of gaze of the user; and at least in part determined by gaze data indicative of the user's gaze point, the gaze data is obtained from a gaze tracking system.

In a specific embodiment, the method includes compressing the pixel data such that the degree of compression is dependent on one of: a distance from the defined point; a direction relative to the defined point; from the defined Points are further increased; and rendering compression is provided.

In a specific embodiment, the method includes: selecting one of a plurality of coding schemes; and encoding the pixel data using the selected coding scheme.

In a specific embodiment, each of the encoding schemes provides a degree of compression, and wherein the method includes selecting the encoding scheme based at least in part on at least one of: a desired degree of compression; and the pixel array The location.

In a specific embodiment, the method includes: determining an encoded code indicating the encoding scheme used; and using the encoded code and the encoded pixel data to generate compressed image data.

In a specific embodiment, the method includes compressing the pixel data by using a coding scheme: applying a conversion to the pixel data to determine a set of frequency coefficients indicating a frequency component of the pixel array; using a bit element Coding scheme selectivity for at least some of these Frequency component encoding, thereby generating a set of encoded frequency coefficients; and using the encoded frequency coefficients to produce compressed image data.

In a specific embodiment, the bit coding scheme defines a number of bits for encoding each of the frequency coefficients, and wherein the frequency coefficients are selectively encoded, such that at least one of: at least some of The encoded frequency coefficients have different number of bits; fewer bits are used to encode frequency coefficients corresponding to higher frequencies; progressively fewer bits are used to encode frequency coefficients corresponding to progressively higher frequencies; Discarding at least one frequency coefficient such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and discarding at least one frequency coefficient corresponding to the higher frequency.

In a specific embodiment, the method includes: selecting one of a plurality of bit coding schemes; and encoding the frequency coefficients in accordance with the selected bit coding scheme.

In a specific embodiment, each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different number of bits to provide different degrees of compression.

In a specific embodiment, the bit encoding scheme is selected based at least in part on at least one of: a desired degree of compression; and the location of the array of pixels.

In a specific embodiment, the frequency components are arranged in a plurality of levels, and wherein each bit coding scheme defines a respective bit to be used to encode the frequency coefficients in each of the plurality of levels Yuan.

In one embodiment, the array is an N x N pixel array resulting in 2 N -1 frequency component levels.

In a specific embodiment, the method includes applying a scaling factor to at least some of the frequency coefficients, such that the scaled frequency coefficients are encoded, the scaling factor The number is used to reduce the strength of each frequency coefficient, and at least one of the following: applying different scaling factors to at least some of the frequency coefficients; applying the same scaling factor to each frequency coefficient; and different The scaling factor is applied to the frequency coefficients in a different channel.

In a specific embodiment, the image data defines a plurality of channels, and wherein the method includes selectively encoding frequency coefficients for each channel.

In a specific embodiment, the pixel data defines an RGB channel, and wherein the method includes converting the RGB channel to a YCbCr channel; and converting the YCbCr channels.

In a specific embodiment, the method includes at least one of: selectively encoding more frequency coefficients for the Y channel than the Cb or Cr channels; selectively encoding frequency coefficients for the YCbCr channels simultaneously; and selectively The frequency coefficients are encoded for the CbCr channels and the Y channel is used.

In one embodiment, the conversion is a two-dimensional (2-D) discrete cosine transform.

In a specific embodiment, the method includes: acquiring pixel data from the image data by buffering image data corresponding to a next n-1 column of pixels of the image; buffering image data of a next n pixels of the next column of pixels; Obtaining pixel data of the next n x n pixel block from the buffered image data; repeating steps b) and c) until pixel data has been obtained from all of the n columns of pixels; and repeating steps a) through d), Until the pixel data has been obtained from each column of pixels of the image.

In a specific embodiment, n is selected according to at least one of: a selected bit coding scheme; a desired degree of compression; and the location of the pixel array.

In a specific embodiment, the method includes: simultaneously selecting an encoding frequency system And generating the compressed image data at least in part by parallel to sequential byte encoding.

In one broad form, aspects of the present invention seek to provide a method of decompressing compressed image data in one or more images forming part of a digital reality content, the method comprising: obtaining compressed image data, The compressed image data represents an array of pixels within the one or more images and is compressed based, at least in part, on a position of the pixel array relative to a defined position within the one or more images. Defining a location at least partially indicating a gaze point of the user; and decompressing the compressed image data based at least in part on the determined location.

In a specific embodiment, the defined location is at least one of: a measured user gaze point; an expected user gaze point; and an offset of the measured user gaze point; Offset to an expected point of gaze of the user; and at least in part determined by gaze data indicative of the user's gaze point, the gaze data is obtained from a gaze tracking system.

In a specific embodiment, the method includes: selecting one of a plurality of decoding schemes; and decoding the pixel data using the selected decoding scheme.

In a specific embodiment, the method includes selecting the decoding scheme based at least in part on at least one of: a desired degree of compression; a location of the pixel array; and an encoding code indicating one of the encoding schemes used, The coded code is determined by the compressed data image.

In a specific embodiment, the method includes decompressing the compressed pixel data using a decoding scheme in accordance with a one-bit encoding scheme in which the number of bits used within each encoded frequency coefficient is defined, Determining a set of encoded frequencies from the compressed image data a rate coefficient; performing bit decoding of the encoded frequency coefficients in accordance with the bit encoding scheme, thereby generating a set of frequency coefficients, wherein at least one frequency coefficient is generated, such that the set of encoded frequency coefficients is less than the set of frequency coefficients; A set of inverse coefficients are applied to the set of frequency coefficients to determine pixel data representing a pixel array within the one or more images.

In a specific embodiment, the bit coding scheme defines the number of bits used to encode each of the frequency coefficients, the bit coding scheme encoding the corresponding higher frequency using fewer bits. a frequency coefficient, and wherein the method includes generating at least some of the frequency coefficients corresponding to a higher frequency domain.

In a specific embodiment, the method includes: selecting one of a plurality of bit coding schemes; and decoding the encoded frequency coefficients in accordance with the selected bit coding scheme.

In a specific embodiment, each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different number of bits to provide a different degree of compression.

In a specific embodiment, the bit encoding scheme is selected based at least in part on at least one of: an encoding code; the bit encoding scheme for generating the compressed image material; and the location of the pixel array.

In a specific embodiment, the frequency components are arranged in a plurality of levels, and wherein each bit coding scheme defines a respective bit to be used to encode the frequency coefficients in each of the plurality of levels Yuan.

In one embodiment, the array is an N x N pixel array resulting in 2 N -1 frequency component levels.

In a specific embodiment, the method includes applying a scaling factor to at least some of the frequency coefficients, such that the scaled encoded frequency coefficients are decoded, the ratio The scaling factor is used to increase the strength of each frequency coefficient, and at least one of them: apply different scaling factors to at least some of the encoded frequency coefficients; apply the same scaling factor to each encoded frequency coefficient And applying a different scaling factor to the encoded frequency coefficients in a different channel.

In a specific embodiment, the image material defines a plurality of channels, and wherein the method includes selectively decoding the encoded frequency coefficients for each channel.

In one embodiment, the compressed image material defines a YCbCr channel, and wherein the method includes: performing an inverse conversion of the YCbCr channels; and converting the converted YCbCr channels to RGB channels.

In a specific embodiment, the method includes at least one of: generating more frequency coefficients to the Cb or Cr channel than the Y channel; simultaneously decoding the encoded YCbCr channels; and decoding the CbCr channels, and The encoded CbCr channel and the Y channel are converted into RGB channels.

In a specific embodiment, the inverse transform is an inverse two-dimensional (2-D) discrete cosine transform.

In a specific embodiment, the method includes decoding the compressed image data at least in part by sequence to parallel byte decoding; and simultaneously selectively decoding the frequency coefficients.

In a specific embodiment, determining a desired degree of compression according to at least one of: a location of the pixel array; a transmission bandwidth of a communication link for transmitting the compressed image data; and transmitting the compressed image The quality of the transmission service of a communication link of the data; the movement of a display device; the image display requirement; the resolution of a target display; the processed one channel; and the error matrix.

In a specific embodiment, the digital reality is at least one of: augmented reality; virtual reality; mixed reality; and telepresence.

In one embodiment, the method is for transmitting the image data from a computing device to a wearable digital reality headset through at least one of: a communication network; and a wireless communication link.

In a broad form, aspects of the present invention seek to provide an apparatus for compressing image data in one or more images forming part of a digital reality content, the apparatus comprising at least one electronic encoder processing device, the at least one An electronic encoder processing device: obtaining pixel data from the image data, the pixel material data representing an array of pixels within the one or more images; determining the pixel array within the one or more images relative to a defined one a location of the location, the defined location at least partially indicating a gaze point of the user; and at least partially compressing the pixel data in accordance with the determined location, such that the degree of compression is dependent on the determined location of the pixel array.

In a broad form, aspects of the present invention seek to provide an apparatus for decompressing compressed image data in one or more images forming part of a digital reality content, the apparatus comprising at least one electronic decoder processing device, The at least one electronic decoder processing device: obtaining compressed image data, the compressed image data representing an array of pixels within the one or more images, and based at least in part on the one or more images relative to A location of the pixel array of a defined location is compressed, the defined location at least partially indicating the gaze point of the user; and the compressed image data is decompressed based at least in part on the determined location.

In a broad form, aspects of the present invention seek to provide a method of compressing image data representing one or more images, the method comprising: obtaining pixels from the image data Data, the pixel material data represents a pixel array within the one or more images; a conversion is applied to the pixel data to determine a set of frequency coefficients indicating a frequency component of the pixel array; using a one-bit encoding scheme Encoding at least some of the frequency coefficients, thereby generating a set of encoded frequency coefficients, wherein the bit encoding scheme defines the number of bits used to encode each of the frequency coefficients, such that the frequency coefficients are selected Sexual encoding: encoding at least some of the encoded frequency coefficients using different number of bits; and discarding at least one frequency coefficient such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and using the encoded frequency coefficients to produce compression Image data.

In a specific embodiment, the frequency coefficients are selectively encoded, such as at least one of the following: defining each of the frequency coefficients to be encoded using a number of bits, independent of the value of the respective frequency coefficients; fewer bits For encoding the frequency coefficients corresponding to the higher frequencies; gradually fewer bits are used to encode the frequency coefficients corresponding to the progressively higher frequencies; and at least one of the frequency coefficients corresponding to the higher frequencies is discarded.

In a specific embodiment, the method includes applying a scaling factor to at least some of the frequency coefficients, such as encoding the scaled frequency coefficients, and wherein at least one of the following: applying different scaling factors to at least some Some frequency coefficients; the same scaling factor is applied to each frequency coefficient; and the scaling factor is used to reduce the strength of each of the frequency coefficients.

In a specific embodiment, the method includes: selecting one of a plurality of coding schemes; and encoding the pixel data using the selected coding scheme.

In a specific embodiment, each of the encoding schemes provides a respective degree of compression, and wherein the method comprises selecting the encoding based at least in part on at least one of: Solution: The degree of compression required; and the location of the pixel array.

In a specific embodiment, the method includes selectively encoding the frequency coefficients in accordance with at least one of: a selection rule; a desired degree of compression; and a location of the array of pixels within the one or more images.

In a specific embodiment, the method includes: selecting one of a plurality of bit coding schemes; and encoding the frequency coefficients in accordance with the selected bit coding scheme.

In a specific embodiment, each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different number of bits to provide different degrees of compression.

In a specific embodiment, the bit encoding scheme is selected based at least in part on at least one of: a selection rule; a desired degree of compression; and the location of the pixel array.

In a specific embodiment, the method includes selecting the bit decoding scheme according to at least one of: a transmission bandwidth of a communication link for transmitting the compressed image data; and a transmission of the compressed image data The quality of the communication service of the communication link; the movement of a display device; the image display requirement; the resolution of a target display; the processed one channel; a position of the pixel array within the one or more images; the one or more The position of the pixel array within the image relative to an observer gaze point of the one or more images; and an error matrix.

In a specific embodiment, the frequency components are arranged in a plurality of levels, and wherein each bit coding scheme defines a respective bit to be used to encode the frequency coefficients in each of the plurality of levels Yuan.

In one embodiment, the array is an N x N pixel array resulting in 2 N -1 frequency component levels.

In a specific embodiment, the method includes: determining the one or more images The gaze point of an observer; the frequency coefficient is selectively encoded at least in part in accordance with the gaze point.

In a specific embodiment, the method includes: determining a distance between the gaze point and a position of the pixel array in the one or more images; and selectively encoding the frequency coefficient according to the distance, such that the larger distance has Fewer frequency coefficients are encoded.

In a specific embodiment, the image data defines a plurality of channels, and wherein the method includes selectively encoding frequency coefficients for each channel.

In a specific embodiment, the pixel data defines an RGB channel, and wherein the method includes converting the RGB channel to a YCbCr channel; and converting the YCbCr channels.

In a specific embodiment, the method includes at least one of: selectively encoding more frequency coefficients for the Y channels than the Cb or Cr channels; selectively encoding more frequency coefficients for the YCbCr channels simultaneously; The compressed image data is generated by encoding the CbCr channels; and using the Y channel.

In one embodiment, the conversion is a two-dimensional (2-D) discrete cosine transform.

In a specific embodiment, the method includes obtaining the pixel data from a video feed.

In a specific embodiment, the method includes: acquiring pixel data from the image data by buffering image data corresponding to the next n -1 column of pixels of the image; and buffering image data of the next n pixels of the next column of pixels; Obtaining pixel data of the next n x n pixel block from the buffered image data; repeating steps b) and c) until pixel data has been obtained from all of the n columns of pixels; and repeating steps a) through d), Until the pixel data has been obtained from each column of pixels of the image.

In a specific embodiment, n is selected according to at least one of: a selection rule; a selected bit coding scheme; and the location of the pixel array.

In a specific embodiment, the method includes: simultaneously selecting a coding frequency coefficient; and generating compressed image data at least in part by parallel to sequential byte encoding.

In a broad form, aspects of the present invention seek to provide an apparatus for compressing image data representing one or more images, the apparatus comprising at least one electronic encoder processing apparatus, the at least one electronic encoder processing apparatus: Obtaining pixel data, the pixel material data representing a pixel array within the one or more images; applying a conversion to the pixel data to determine a frequency coefficient indicating a frequency component of the pixel array; using one A bit coding scheme selectively encodes at least some of the frequency coefficients, thereby generating a set of encoded frequency coefficients, wherein the bit coding scheme defines the number of bits used to encode the equal frequency coefficients, and wherein When the frequency coefficients are selectively encoded: at least some of the encoded frequency coefficients have different number of bits; discarding at least one frequency coefficient such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and using the encoded frequency coefficients to generate Compressed image data.

In one embodiment, the apparatus includes an encoder input buffer for receiving the image data, and an encoder output buffer for storing the compressed image data.

In a specific embodiment, the device includes an encoder input buffer that buffers image data corresponding to the next n -1 column of pixels of the image; buffers the next n of the next column of pixels Image data of the pixel, allowing the at least one encoder processing device to acquire pixel data of the next n x n pixel block from the buffered image data; repeating step b) until pixel data has been obtained from all of the n columns of pixels And repeat steps a) and b) until the pixel data has been obtained from each column of pixels of the image.

In a specific embodiment, the apparatus includes an encoder transmitter that transmits the image data from the encoder output buffer.

In a specific embodiment, the at least one encoder processing device comprises: a field programmable gate array; a dedicated integrated circuit and a graphics processing unit.

In a specific embodiment, the pixel data defines a plurality of channels, and wherein the device includes at least one of a respective processing device of each channel and a parallel processing device for processing each channel simultaneously.

In a specific embodiment, the pixel data defines an RGB channel, and wherein the device: converts the RGB channel to a YCbCr channel; and selectively encodes the YCbCr channels using processing means.

In a specific embodiment, the pixel data defines an RGB channel, and wherein the device: converts the RGB channels into CbCr channels using a YCbCr processing device; decodes the CbCr channels using at least one respective processing device; The Y channel is converted from the YCbCr processing device to an output buffer using a delay block.

In one embodiment, the apparatus includes an encoder that communicates with a decoder by wireless communication to allow image data to be converted between the encoder and the decoder in a compressed image format.

In a specific embodiment, the encoder is at least one of a processing system coupled to a suitable program editor and a portion thereof.

In a specific embodiment, the decoder is coupled to at least one of a wearable display device and a portion thereof.

In a specific embodiment, the encoder and the decoder communicate to exchange at least The next one: compressed image data; mobile data indicating a movement of the display device; at least part of control data for controlling the display device; input data indicating a user input instruction; and gaze data indicating an observer's gaze point And sensor data from sensors associated with a wearable display device.

In a broad form, aspects of the present invention seek to provide a method of decompressing compressed image data representing one or more images, the method comprising: acquiring compressed image data; each encoded frequency is defined by definition a one-bit encoding scheme for the number of bits used in the coefficient, determining a set of encoded frequency coefficients from the compressed image data; performing bit decoding of the encoded frequency coefficients according to the bit encoding scheme, thereby generating a Generating a frequency coefficient, wherein at least one frequency coefficient is generated, such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and applying a reverse transform to the set of frequency coefficients to determine a pixel array representing the one or more images Pixel data.

In a specific embodiment, the method includes: selecting one of a plurality of decoding schemes; and decoding the pixel data using the selected decoding scheme.

In a specific embodiment, the method includes selecting the decoding scheme based at least in part on at least one of: a selection rule; a desired degree of compression; a location of the pixel array; and an encoding indicating one of the encoding schemes used Code, the code code is determined by the compressed data image.

In a specific embodiment, the bit encoding scheme uses fewer bits to encode frequency coefficients corresponding to higher frequencies, and wherein the method includes generating at least some of the frequency coefficients corresponding to the higher frequency domain.

In a specific embodiment, the method includes applying a scaling factor to At least some of the frequency coefficients, such that the scaled frequency coefficients are converted, and wherein at least one of: applies a different scaling factor to at least some of the encoded frequency coefficients; applying the same scaling factor to each The encoded frequency coefficient; and the scaling factor is used to increase the strength of each of the encoded frequency coefficients.

In a specific embodiment, the method includes: selecting one of a plurality of bit coding schemes; and decoding the encoded frequency coefficients in accordance with the selected bit coding scheme.

In a specific embodiment, each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different number of bits to provide different degrees of compression.

In a specific embodiment, the bit coding scheme is selected based at least in part on at least one of: an encoding code; a selection rule; the bit encoding scheme for generating the compressed image material; and the pixel array The location.

In a specific embodiment, the selection rule is based on at least one of: a transmission bandwidth of a communication link for transmitting the compressed image data; and a transmission service quality of a communication link for transmitting the compressed image data; a display device movement; an image display requirement; a target display resolution; a processed one channel; a position of the pixel array within the one or more images; the pixel array within the one or more images One of the viewer's gaze points relative to the one or more images; and an error matrix.

In a specific embodiment, the method includes determining an observer's gaze point for the one or more images; selectively decoding the encoded frequency coefficients based at least in part on the gaze point.

In a specific embodiment, the method includes: determining a distance between the gaze point and a position of the pixel array in the one or more images; and encoding the gaze according to the distance The frequency coefficient is selectively decoded, so that a larger distance produces more frequency coefficients.

In a specific embodiment, the frequency components are arranged in a plurality of levels, and wherein each bit coding scheme defines a respective bit to be used to encode the frequency coefficients in each of the plurality of levels Yuan.

In one embodiment, the array is an N x N pixel array resulting in 2 N -1 frequency component levels.

In a specific embodiment, the image material defines a plurality of channels, and wherein the method includes selectively decoding the encoded frequency coefficients for each channel.

In one embodiment, the compressed image material defines a YCbCr channel, and wherein the method includes: performing an inverse conversion of the YCbCr channels; and converting the converted YCbCr channels to RGB channels.

In a specific embodiment, the method includes at least one of: generating more frequency coefficients to the Cb or Cr channel than the Y channel; simultaneously decoding the encoded YCbCr channels; decoding the CbCr channels, and The decoded CbCr channel and the Y channel are converted into RGB channels.

In a specific embodiment, the inverse transform is an inverse two-dimensional (2-D) discrete cosine transform.

In a specific embodiment, the method includes using the pixel data to generate a video feed.

In a specific embodiment, the method includes decoding the compressed image data at least in part by sequence to parallel byte decoding; and simultaneously selectively decoding the frequency coefficients.

In a specific embodiment, the digital reality is at least one of: augmented reality; virtual reality; and mixed reality.

In a specific embodiment, the method is for displaying image data in a wearable digital reality wearing group by receiving the compressed image data from a computing device through at least one of: a communication network; And a wireless communication link.

In a specific embodiment, the method is for at least one of: transmitting virtual reality video material; and wirelessly transmitting virtual reality video material.

In a broad form, aspects of the present invention seek to provide an apparatus for decompressing compressed image data representing one or more images, the apparatus comprising at least one electronic decoder processing device, the at least one electronic decoder processing Means: acquiring compressed image data; determining a set of encoded frequency coefficients from the compressed image data according to a one-bit encoding scheme defining the number of bits used in each encoded frequency coefficient; encoding according to the bit encoding a scheme for performing bit decoding of the encoded frequency coefficients, thereby generating a set of frequency coefficients, wherein at least one frequency coefficient is generated, such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and applying a reverse to the set of frequency coefficients Converting to determine pixel data representing a pixel array within the one or more images.

In one embodiment, the apparatus includes a decoder input buffer for receiving the compressed image data, and a decoder output buffer for storing the image data.

In one embodiment, the device includes a decoder transceiver that receives the compressed image data and provides the compressed image data to the input buffer.

In a specific embodiment, the at least one decoder processing device comprises: a field programmable gate array; a dedicated integrated circuit and a graphics processing unit.

In a specific embodiment, the compressed image data defines a plurality of channels, And wherein the device includes at least one of a respective processing device of each channel and a parallel processing device for processing each channel simultaneously.

In one embodiment, the compressed image material defines a YCbCr channel, and wherein the device: decodes the CbCr channels using at least one processing device; and converts the decoded YCbCr channels into RGB channels.

In a specific embodiment, the compressed image material defines a YCbCr channel, and wherein the device: decodes the CbCr channels using a processing device; and converts the decoded CbCr channels and the Y channel into an RGB processing device RGB channel; and converting the Y channel from a decoder input buffer to the RGB processing device using a delay block.

In one embodiment, the apparatus includes a decoder for communicating with an encoder in a wireless communication manner to allow image data to be transmitted as compressed image data between the encoder and the decoder.

In one embodiment, the decoder is at least one of a computer system coupled to a suitable program editor and a portion thereof.

In a specific embodiment, the decoder is coupled to at least one of a wearable display device and a portion thereof.

In a specific embodiment, the decoder and the encoder communicate to exchange at least one of: compressed image data; mobile data indicating movement of a display device; control data at least partially used to control the display device; The input data of the input command; the gaze data indicating an observer's gaze point; and the sensor data from the sensor associated with a wearable display device.

In a broad form, aspects of the invention seek to provide a representative Or a method for compressing image data of a plurality of images, the method comprising: obtaining pixel data from the image data, the pixel material data representing a pixel array within the one or more images; determining a coding scheme; using the coding scheme Encoding the pixel data; determining an encoding code indicating the encoding scheme used; and using the encoding code and the encoded pixel data to generate compressed image data.

In a specific embodiment, the method includes determining the encoding scheme by using at least one of: an image type according to the image data; an encoding scheme indication received from an image data source; and utilizing analyzing at least the image data and One of the pixel data; and a compression requirement according to at least one of: a total amount of compression; a resulting image quality; and a compression delay.

In a specific embodiment, the method includes analyzing the pixel data to determine whether the pixel array is at least one of: a gradient; a boundary; and a single color.

In a specific embodiment, the method includes at least one of: if the pixel array is a solid color, replacing the pixel array with an encoding code representing the solid color; if the pixel array is a gradient, the method includes using The pixel material is encoded according to a method of another aspect of the present invention; and the pixel material is decoded using a method in accordance with another aspect of the present invention.

In a broad form, aspects of the present invention seek to provide an apparatus for compressing image data representing one or more images, the apparatus comprising at least one electronic encoder processing apparatus, the at least one electronic encoder processing apparatus: Obtaining pixel data, the pixel material data representing a pixel array within the one or more images; determining an encoding scheme; encoding the pixel data using the encoding scheme; determining an encoding indicating the encoding scheme used a code; and using the encoded frequency coefficient to generate compressed image data.

In a broad form, aspects of the present invention seek to provide a method of decompressing compressed image data representing one or more images, the method comprising: acquiring compressed image data; determining from the compressed image data An encoding code; determining an encoding scheme using the encoding code; and decoding the at least partially compressed image data using the encoding scheme to determine pixel data representing a pixel array within the one or more images.

In a specific embodiment, the method includes at least one of: replacing an encoded code with a solid color pixel array; and using the method according to another aspect of the present invention, decoding the compressed image data into a gradient one a pixel array; and using a method in accordance with another aspect of the present invention to decode the compressed image material.

In a broad form, aspects of the present invention seek to provide an apparatus for decompressing compressed image data representing one or more images, the apparatus comprising at least one electronic decoder processing device, the at least one electronic decoder processing Means: acquiring compressed image data; determining an encoding code from the compressed image data; determining an encoding scheme by using the encoding code; and decoding the compressed image data using the encoding scheme to determine whether to represent the one or more Pixel data of a pixel array within an image.

It is to be understood that the invention in its broader aspects and its various features may be used in combination, interchangeable and/or independently.

Steps 100-170‧‧

200‧‧‧Display equipment on wearable devices

210‧‧‧Processing system

220‧‧‧Encoder

230‧‧‧Decoder

240‧‧‧HMD

250‧‧‧ Controller

260‧‧‧Wireless communication link

300‧‧‧Virtual reality systems are combined for Device for compressing and decompressing image data

310‧‧‧Processing system

311‧‧‧Microprocessor

312‧‧‧ memory

313‧‧‧Input/output devices

314‧‧‧ external interface

315‧‧‧ busbar

320‧‧‧Encoder

321‧‧‧Encoder input buffer

322‧‧‧Encoder processing device

323‧‧‧Encoder output buffer

324‧‧‧ transceiver

325‧‧‧individual data buffer

330‧‧‧Decoder

331‧‧‧Decoder input buffer

332‧‧‧Decoder processing device

333‧‧‧Decoder output buffer

334‧‧‧ transceiver

335‧‧‧individual data buffer

340‧‧‧ display device

341‧‧‧Microprocessor

342‧‧‧ memory

343‧‧‧Input/output devices

344‧‧‧ Sensor

345‧‧‧ display

346‧‧‧ external interface

347‧‧‧ busbar

360‧‧‧Short range radio communication

400-448‧‧‧Steps

531-533‧‧‧frequency factor

541‧‧‧Brightness channel

542‧‧‧Color channel

543‧‧‧Color channel

600-630‧‧ steps

701‧‧‧viewable section

702‧‧‧Unvisible part

800-860‧‧‧Steps

900-928‧‧‧Steps

1000-1040‧‧‧Steps

Referring to the drawings, an example of the invention will be described, in which: the first figure is a flowchart of an example of a method for compressing and subsequently decompressing image data; the second figure is for displaying an image on a wearable device One device A first exemplary diagram of a second example; a second exemplary diagram of a device for displaying an image on a wearable device; and a third schematic diagram of a virtual reality system incorporating a device for compressing and decompressing image data A schematic diagram of a specific example; the fourth through fourth graphs are flowcharts of a specific example of a method for compressing and subsequently decompressing image data; the fifth graph is a schematic diagram illustrating the encoding processing aspect; A flow chart of an example of a method for selecting a one-bit coding scheme; a seventh diagram of an example of an image to be encoded; and an eighth diagram of an encoding/decoding scheme for compressing image data and decompressing compressed image data A flow chart of an example; ninth A and ninth B are flowcharts of a specific example of a method of compressing and subsequently decompressing image data using a selected encoding/decoding scheme; and the tenth figure is for compression And a flow chart of a further example of a method for subsequently decompressing image data.

An example of a method for compressing and decompressing image data will be described with reference to the first figure.

For purposes of illustration, it is assumed that the process is performed, at least in part, using one or more electronic processing devices. In one example, an individual processing device is used to compress and decompress the image data, allowing compressed image data to be transferred between the two processing devices, although this is not required and Optionally, the same processing device can be used to compress and decompress the image data.

The processing device can form part of a separate processing system, such as a computer system, a computer server, a client device including a mobile phone, a portable computer, a display device such as a wearable or head mounted display, or It can be in the form of a separate module that is attached to such a device.

The image material typically represents one or more images and, in one example, represents a series of images to be displayed on respective display devices. As will be appreciated from the following description, in one particular example, the image data is a series of images suitable for remote display for a source, such as a virtual or augmented reality in which the image is displayed on a wearable display. In graphics applications, and/or in telepresence applications where images are displayed from a remotely controllable system such as a drone-equipped camera.

In this example, at step 100, pixel data is obtained from the image data, the pixel material data representing a pixel array within the one or more images. The pixel data can be obtained in any suitable manner, depending on the format of the image material. In one example, a simple selection of a particular byte sequence within the image data can be achieved. The array of pixels typically corresponds to a set of pixels, such as an 8x8 pixel block within one of the images, although other pixel arrays can be used.

At step 110, a conversion is applied to the pixel data to determine a set of frequency coefficients indicative of the frequency components of the pixel array. Therefore, the conversion is typically a frequency conversion, such as a Fourier transform, etc., and in one example is a 2D DCT (Discrete Cosine Transform). This conversion can be applied in any suitable manner, for example using known conversion techniques, but in one example in a high parallel manner, thereby reducing processing time.

At step 120, at least some of the frequency systems are used using a one-bit encoding scheme Number selective coding, thereby generating a set of encoded frequency coefficients. The bit coding scheme defines a number of bits to be used to encode each frequency coefficient, after the frequency coefficients are selectively encoded, at least some of the encoded frequency coefficients have different number of bits, and at least one frequency coefficient Abandoned, such that the set of coded frequency coefficients is less than the set of frequency coefficients.

This processing can be accomplished in any suitable manner and can include discarding some of the frequency coefficients and then encoding the remaining frequency coefficients using different number of bits, thereby reducing the number of bits required to encode the frequency coefficients. Alternatively, the process should include encoding some of the frequency coefficients with zero bits, thereby effectively discarding the respective frequency coefficients as part of the encoding step.

The particular frequency components that have been discarded will vary with the preferred implementation, and generally higher frequency components will be discarded due to their lower intensity and due to their sharp transitions within the image, meaning that they are for overall image quality. Less contribution. This allows discarding higher frequency component coefficients without significantly adversely affecting the perceived image quality. In addition to discarding the frequency components corresponding to the higher frequencies, the process can encode the frequency coefficients of the higher frequency components with fewer bits, thereby reducing the total number of bits needed to encode the frequency coefficients.

Similarly, when the frequency coefficients are encoded with different number of bits, their execution is independent of the actual value of the frequency coefficient, but is performed based on an understanding of the expected strength of the frequency coefficient. For example, the frequency coefficient strength at lower frequencies is usually larger, so it is usually coded with more bits, while the frequency coefficient strength at higher frequencies is usually smaller, so it is coded with fewer bits. This allows the values of the frequency coefficients to be encoded without loss of information.

Once the encoding has been performed, the encoded image coefficients can be used to generate compressed image data at step 130, for example, by performing a one-tuple stream, including the sequence of encoded frequency coefficients, optionally with additional Information, such as flags or other markers to identify The beginning of new images and so on.

Thus, the above process allows for the use of a one-bit coding scheme that discards at least some of the frequency coefficients and encodes the remaining coefficients using different number of bits, for example, based on the strength of the frequency coefficients, selectively encoding the frequency coefficients to establish a compressed video material. Thus, a smaller number of bits can be used to encode a smaller intensity factor without any loss of information.

It should be noted that this approach should be contrasted with code replacement techniques, such as Huffman encoding, where values replace short codes. Conversely, in this example, the value is still encoded, although the number of bits appropriate for the expected strength of the value is used, so if the frequency coefficient value is expected to be no more than seven, then this can be encoded as a three-dimensional character, so Instead of using the default octet character "00000110", the six will be coded as "110". In contrast, if the value of the frequency coefficient is expected to be as high as sixty-three, a six-bit character should be used. For example, twenty should be coded as "010100". In the event that the value is higher than the number of available bits, the maximum value available for the defined number of bits should be used, resulting in a loss of precision in the resulting compressed image data.

As such, the bit encoding scheme uses information about the expected size of the frequency coefficient values to define the number of bits that should be used. A less aggressive bit-encoding scheme will use more bits, resulting in less compression, but with higher resolution, while a more aggressive bit-coding scheme will use fewer bits, thus providing more compression. But the resolution is low.

In any event, by using a one-bit encoding scheme, which defines the number of bits used to encode each frequency coefficient, this allows the same scheme to be used when decompressing the compressed image material, and then allows proper decompression to be performed. At the same time, it is allowed to set the bit coding scheme used, and the compression for the current situation is optimized.

In this regard, at step 140, according to the bit coding scheme, from the The compressed image data determines a set of encoded frequency coefficients. In particular, by selecting the next number of bits that make up the next frequency coefficient, the information about the number of bits used to encode each frequency coefficient allows the received compressed image data to be divided into the encoded frequency coefficients. .

At step 150, selective bit decoding of the encoded frequency coefficients is performed in accordance with the bit encoding scheme, thereby generating a set of frequency coefficients. In this regard, it is performed to convert each encoded frequency coefficient into a frequency coefficient and additionally generate a frequency coefficient to be discarded during the encoding process. In particular, it is typically performed to generate a frequency coefficient that has a null value, thereby again establishing an entire set of frequency coefficients.

Next, an inverse transformation is applied to the set of frequency coefficients to determine pixel data representing a pixel array within the one or more images. In particular, it is usually in the form of inverse frequency conversion, such as inverse Fourier transform, 2D DCT, and the like.

Thus, the above process allows the encoded video coefficients to be encoded by selectively encoding the frequency coefficients using a one-bit encoding scheme, and then encoding the encoded frequency coefficients using the same bit encoding scheme. Still further, the bit encoding scheme used can be adapted and can depend on a wide range of criteria, such as the nature of the encoded image material, the particular channel encoded, and the like. This allows the bit coding scheme to be applied, whereby the maximum amount of compression can be achieved.

In addition to the above advantages, the scheme can be implemented in a highly parallel manner, for example allowing simultaneous encoding of each frequency coefficient. This in turn enables the processing to be performed quickly, thereby reducing the delay time, which is quite important in many applications, such as virtual reality applications, in which the display device is moved to create an image and must be quickly transferred to the display device. display.

Some further features will be described at this time.

In one example, the bit coding scheme uses a smaller number of bits to encode Corresponding to higher frequency frequency coefficients, since the higher frequency components have less intensity, which means that a lower number of bits is needed to accurately encode higher frequency coefficients, lower frequencies. In one example, a decreasing number of bits is used to encode frequency coefficients corresponding to progressively higher frequencies. In this case, the frequency coefficients at successive higher frequencies should have a number of bits with a number of bits equal to or lower than the frequency coefficient of a previous lower frequency. Similarly, the method can include discarding at least some of the frequency coefficients corresponding to the higher frequencies, which tends to have less impact on the perceived image quality. It should also be appreciated that for some of these lower frequency coefficients, all of the bits required to encode the coefficients may be retained, and in some examples may exceed eight bits.

In one example, the method includes applying a scaling factor to at least some of the frequency coefficients, such that the scaled frequency coefficients are encoded. In this regard, scaling is used to reduce the strength of the frequency coefficients such that the coefficients can be encoded using fewer bits. A similar scaling factor can be applied when performing decompression, whereby the individual frequency components are scaled back to their original intensity. During this process, numerical rounding is typically performed such that the scaled frequency components are integer values, or have a finite amount of valid data, thereby minimizing the number of bits used to encode the coefficients. It should also be understood that the accuracy of these re-established frequency components will decrease when executed, but the effect of this effect on the resulting image quality is negligible.

In one example, the same scaling factor is applied to each frequency coefficient. This is particularly advantageous because it reduces the computational burden when performing the scaling. In particular, this allows a single scaling factor to be read from a memory, such as a scratchpad, or to allow writing within a logical configuration, thereby making the scaling of the frequency coefficients faster. However, this is not required, and different scaling factors can be applied to different frequency coefficients, such as scaling the higher frequency coefficients in larger proportions.

In one example, the method includes applying a scaling factor to the frequency components to determine a scaled frequency component, selecting one or more scaled frequency components in accordance with a selection criterion, and following a one-bit encoding scheme The bit encoding of the selected scaled frequency component is performed to generate compressed image data, thereby limiting the encoding of each selected scaled frequency component. However, this is not required and other methods should be used, such as performing scaling or the like after discarding some of these frequency coefficients.

In one example, the method can include selecting one of a plurality of coding schemes and encoding the pixel data using the selected coding scheme, which allows for different coding schemes to be selected based on factors such as the degree of compression desired. . As such, each of the encoding schemes can provide a degree of compression, such as by using a different compression scheme, or using one of a plurality of different bit encoding schemes. In this latter case, each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different number of bits to provide different degrees of compression.

Depending on a range of coefficients, such as selection rules, desired degree of compression, and/or position of the pixel array within the one or more images, the particular encoding scheme used may be selected and then used to provide the desired compression, More details will be given below.

Similarly, in one example, the method generally includes selectively encoding frequency coefficients in accordance with a selection rule, a desired degree of compression, and/or a position of the pixel array within the one or more images. In this regard, the selection rules can be used to define which frequency coefficients to encode, and/or which particular bit encoding scheme to use, which actually achieves the same end result and should therefore be considered an equivalent procedure.

For example, the selection rules can be used to select a subset of the frequency coefficients for Encoding is then encoded using the bit encoding scheme. Alternatively, a one-bit encoding scheme is selected in accordance with a selection rule, the at least some of the frequency coefficients are encoded in zero bits using the bit encoding scheme, and the frequency coefficients are then encoded in accordance with the bit encoding scheme.

In both cases, the use of a selection rule allows dynamic execution of the selective bit coding, such that the selected frequency coefficients and/or the number of bits used to encode the selected frequency coefficients can be adjusted according to circumstances. When the compressed image data is decompressed, a similar selection rule can be used, thereby allowing the bit encoding scheme and/or frequency coefficient discarding to be dynamically performed according to the particular environment, to determine that the compressed data can be decompressed accurately.

In one example, this allows for consideration of a number of different factors, such as the transmission bandwidth of a communication link used to transmit the compressed image data, the transmission service quality of a communication link used to transmit the compressed image data, and display. Movement of the device, image display requirements, target display resolution, processed image channel, location of the pixel array within the one or more images, or an observer gaze point for the one or more images The location of the array of pixels within the one or more images. For another alternative, an error metric indicating error and/or data transmission within the decompressed image may be used to control the degree of compression used. It should also be understood that these configurations can be used to dynamically adjust the degree of compression, such as by using the bit coding scheme used to compress the image data, for example, if the compression exceeds a critical limit, compression can be reduced, and if the available transmission bandwidth is reduced, Then you can increase the compression. This ability to dynamically adjust compression helps optimize this compression to achieve the best possible image quality for the current environment.

For example, the relative quality of some parts of an image may not be as important as the other parts. In the case of virtual reality, since the image of the display lens is distorted, the peripheral portion of the image is usually not displayed to the user. Therefore, these image parts should use an effective zero quality The code can greatly reduce the amount of compressed image data without losing any image quality of the visible image.

In another example, particularly in virtual reality applications, this can be performed in accordance with an observer gaze point. In this example, this involves determining an observer gaze point for the one or more images and selectively encoding the frequency coefficients based at least in part on the gaze point. In particular, this may involve determining the distance between the gaze point and the location of the pixel array within the one or more images, and selectively encoding the frequency coefficients in accordance with the distance such that the farther the distance, the less the frequency coefficients are encoded. In this way, an analysis of which portion of the image the viewer views can be performed, such as using eye tracking techniques or the like, and then encoding the portion of the image near the gaze point with a higher quality. In this respect, an observer's gaze to the surrounding area is usually reduced, so that the image quality is generally less noticeable. Therefore, encoding images near the observer's gaze point with higher quality will result in an overall poor quality image being perceived by the observer as having a general level of quality.

In one example, the frequency components are arranged in a plurality of levels, each bit encoding scheme defining a respective number of bits to be used to encode the frequency coefficients in each of the plurality of levels. Similarly, or a selection rule defined by which hierarchical frequency coefficient should be encoded. As such, the selection of the bit coding scheme and/or frequency coefficients can be defined in accordance with the respective levels.

In one example, the pixel array is an N x N pixel array that produces 2 N -1 order frequency components, although it will be appreciated that this will depend on the particular implementation.

In one example, in addition to performing the above-described corrupted compression, an additional non-destructive compression step can be performed. This typically involves parsing a sequence of bytes, identifying subsequences containing some consistent sets of bytes, and replacing the subsequences with values indicating the identical sets of bytes and some consistent bytes within the subsequence The code. In one example, when consistent bytes When the subsequence includes three or more bytes, the code includes two bytes, although it should be understood that other suitable coding schemes should be available.

Although code replacement, commonly referred to as run length encoding, can be performed on any sequence of bytes, in one example, the sequence of bytes is the stream of bits formed from the encoded frequency coefficients. In this regard, there are typically a number of such coded frequency coefficients having a value of zero, indicating that when the bit stream formed from the encoded frequency coefficients is analyzed as a sequence of bytes, a number of zero bits are frequently present within the sequence. Tuple. Therefore, by replacing these with a code, this allows for a reduction in the number of bytes.

In one example, the image data defines a plurality of channels, and wherein the method includes selectively encoding frequency coefficients for each channel. By encoding different channels separately, this allows different coding of different channels, for example using different bit coding schemes, or discarding different frequency coefficients. In addition, separate encoding channels allow the channels to be encoded at the same time, which greatly helps to shorten the encoding execution time and therefore the encoding delay.

In one example, the pixel data defines RGB channels, and the method includes converting the RGB channels to luminance and chrominance channels YCbCr and transforming the YCbCr channels. In this respect, the human eye feels differently for the luminance and chrominance channels, allowing the chrominance channel to be encoded with a greater degree of compression, so the quality is lower compared to the luminance channel, but the perceived quality is not compromised. However, it should be understood that this is not required and processing may additionally be performed within the RGB channels, in which case color conversion is not necessary.

Thus in this example, the method can include selectively encoding more frequency coefficients for the Y channel than the Cb or Cr channel, and similarly can include selectively using the bits over the Cb and Cr channels for the Y channel Coding frequency coefficient.

In a further example, where the pixel data defines an RGB channel, the method can include converting the RGB channels to a YCbCr channel and utilizing the CbCr channels to encode the compressed image data. In fact, in this example, the Y channel is not effectively coded, indicating that the complete information contained in the luminance channel is retained. This is particularly useful in certain coding scenarios, such as when encoding a progressive array of pixels, which helps preserve color variations, thus improving image quality while only causing a slight reduction in compression.

As mentioned above, the different channels can be encoded simultaneously. In addition, each of the frequency coefficients can be encoded simultaneously. In this case, the method of generating compressed image data typically includes simultaneously performing serial string encoding, such that the frequency coefficients are serialized into a one-tuple stream and then byte encoded.

In one example, the method for obtaining pixel data from image data includes buffering image data corresponding to a next n -1 pixel column in the image, buffering image data of a next n pixels of the next pixel column, and The pixel data of the next n x n pixel block is obtained from the buffered image data. Repeat this process until the pixel data has been acquired from all n columns of pixels, and the program is repeated for the next n columns by buffering the image data corresponding to the next n -1 pixel column in the image, and the next pixel column The image data buffer of the next n pixels, and the pixel data of the next n x n pixel block is obtained from the buffered image data.

Thus, it can be understood from this that the process does not require the entire image to be buffered, but only the n-1 column pixels and the further n pixels from the next column are buffered before starting the process. There are two main shocks, in other words, shortening the processing time, which in turn causes a significant reduction in latency while reducing overall memory requirements. The value of n is usually an integer and can be set according to factors such as selection rules, degree of compression required, position of the pixel array, and the like. In one example n = 8, but this is not necessary and any value can be used.

Although image data can be obtained from any source, in one example, the method includes obtaining the pixel data from a video feed, such as a series of images to be displayed. In one example, the method is for transmitting virtual reality video material and, in a particular example, for wirelessly transmitting virtual reality video material.

The above process can also be used to provide images that form part of any digital reality content, including augmented reality, virtual reality, mixed reality, telepresence, and the like.

In one example, the method is for receiving, by one of the at least one communication network and a wireless communication link, the compressed image data from a computing device, displaying the image in a wearable digital reality wearing set data. This may include wirelessly transmitting the compressed image from a computer to other similar devices, or may include transmitting the compressed image to a native device from a cloud computing environment, such as a head-mounted smart phone, allowing for cloud computing to be used. Perform image creation. Examples of suitable connections, including a wired GB internet, streaming to mobile phones, such as via a mobile communication network, such as a 3G, 4G or 5G network, over a wired connection to a bonded HMD, or via a wireless connection Transfer to unbound HMD, etc.

It should be appreciated that a similar approach can be used when decompressing the compressed image material.

For example, the same bit coding scheme can be used, such that the bit coding scheme uses fewer bits to encode frequency coefficients corresponding to higher frequencies. In this case, discarding at least some of the frequency coefficients corresponding to the higher frequencies during compression must be regenerated during decompression, typically null, allowing subsequent inverse transformations to be applied, as described in more detail below.

Typically the method includes applying a scaling factor to at least some of the frequency coefficients, such that the scaled frequency coefficients are converted. Again, the same scale scaling factor set Each frequency coefficient is used, thereby increasing the intensity of each frequency coefficient, thereby reversing the scaling performed during compression, and again producing an approximation of the original frequency coefficient strength.

The method can include selecting one of a plurality of decoding schemes and decoding the pixel data using the selected decoding scheme. In this case, the decoding scheme can be selected using a selection rule, a desired degree of compression, or a location of the pixel array, similar to the encoding scheme selection described above with respect to image compression. Alternatively, this may be performed in accordance with an encoding code indicating the encoding scheme used, the encoding code being determined by the compressed image material.

Again, the decoding scheme can be different than that described above, but one of a plurality of different bit encoding schemes can be used as well, and the encoded frequency coefficients are decoded in accordance with the selected bit encoding scheme. In this case, each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different numbers of bits to provide different degrees of compression.

Similarly and effectively equivalently, the method can include selectively decoding the encoded frequency coefficients in accordance with a selection rule, in particular by generating the encoded frequency coefficients in accordance with a selection rule and decoding the encoded frequencies in accordance with the bit encoding scheme coefficient.

During compression, the selection rules generally depend on a range of factors, such as one or more transmission bandwidths used to transmit the compressed image data, and a communication link for transmitting the compressed image data. Transmission service quality, movement of the display device, image display requirements, target display resolution, processed one channel, location of the pixel array within the one or more images, or one of the one or more images The viewer gaze at the location of the array of pixels within the one or more images of the point.

For example, this may involve determining an observer gaze point for the one or more images and selectively decoding the encoded frequency coefficients based at least in part on the gaze point. This usually passes The distance between the gaze point and the position of the pixel array in the one or more images is determined, and the encoded frequency coefficient is selectively decoded according to the distance, such that the further the distance, the more frequency coefficients are generated. However, it should be appreciated that the implementation of the selection rules can be implemented in any suitable manner in accordance with the preferred implementation.

The frequency components are typically arranged in a plurality of levels, using the bit coding scheme, which defines respective bits of the frequency coefficients to be used to encode frequency components corresponding to individual ones of the plurality of levels Number, or use these selection rules, which are defined by which hierarchical frequency coefficient should be generated.

In the event of also performing non-destructive encoding, the method typically includes identifying a code within the sequence of bytes and replacing the code with a subsequence containing some consistent sets of bytes. In this case, the code typically indicates the value of the identical byte and some consistent bytes within the subsequence. Again, the subsequence typically includes three or more bytes, and the code must contain two tuples, although other suitable configurations can be used. This process is typically performed on the compressed image data, which is used to generate the bit stream, which is then used to establish the encoded frequency coefficients.

As previously explained, the image data typically defines a plurality of channels, and the encoded frequency coefficients selectively decode each channel individually. The channels typically include a YCbCr channel, and the method includes performing an inverse transform of the YCbCr channels and converting the transformed YCbCr channels to RGB channels. Typically, the inverse transform is an inverse 2-D discrete cosine transform, although other suitable transforms can be used. It should also be appreciated that if the Y channel is not yet encoded, as described above, the method can include decoding the CbCr channels and then converting the decoded CbCr channels and the Y channels to RGB channels.

In an example of compressing the image data, the method generally includes generating more frequency coefficients to the Cb or Cr channels than the Y channel. The method can also include simultaneously decoding the encoded The YCbCr channel also selectively performs bit decoding on individual frequency components at the same time. In the case, the compressed image data is decoded at least partially by serial to parallel byte decoding, effectively separating the incoming byte stream into individual bits. The frequency encodes the frequency components and then decodes them simultaneously.

The decompressed data can also be further processed, such as using a deblocking filter for sharpness between macroblocks when using block coding techniques and the like. The edges are smooth. This in turn allows for a higher degree of compression while avoiding a corresponding reduction in image quality.

In a further example, the method is performed by a separate hardware configuration, for example, the compressed image data may be executed by an encoder including an electronic encoder processing device, wherein the device obtains pixel data from the image data, and executes one Frequency transforming, selectively encoding at least some of the frequency coefficients using a one-bit encoding scheme, and using the encoded frequency coefficients to produce compressed image data.

Similarly, decompressing the compressed image material is performed using a decoder including an electronic decoder processing device, wherein the device obtains compressed image data, determines a set of encoded frequency coefficients from the compressed image data, The bit encoding scheme performs bit decoding of the encoded frequency coefficients and applies an inverse transform to the set of frequency coefficients to determine pixel data representing a pixel array within the one or more images.

In one example, the device (200) includes an encoder and a decoder for wireless communication that allows image data to be transferred between the encoder and the decoder in a compressed image format. In one particular example, this can be used to provide a wearable display device, such as a wireless communication between an HMD and a processing system. This example will be explained with reference to the second A diagram.

In this example, it is suitable for program editing computer systems, game consoles, etc. A processing system 210 is adapted to generate display content on an HMD 240. Processing system 210 typically accomplishes this by receiving sensor data from the HMD regarding the HMD gesture and selectively inputting data from one or more individual controllers 250. Processing system 210 then generates content based on the sensor and/or input data, typically in the form of video material, which can be output from a video card or the like. The video data is transmitted to an encoder 220, which compresses the video data to encode the video data, and then wirelessly transmits the compressed video data to the decoder 230 via a wireless communication link 260. The decoder 230 decodes the compressed image data and provides the resulting video material to the HMD for display.

It should be appreciated that this configuration allows existing computer systems, gaming machines, etc., as well as HMDs 210, 240 to be connected through wireless connection 260, thus avoiding the need for a wired connection between processing system 210 and HMD 240. Thus, for example, a user can wear an HMD and associated decoder and then connect the encoder to their computer system or gaming machine to form a wireless HMD configuration. This can be used to convert a traditional tethered headset into a wireless configuration.

However, this is not required, or processing system 210 and HMD 240 may be configured to include integrated encoder and decoder hardware, allowing these to communicate over direct wireless connection 260, as shown in FIG. For example, the encoder can be provided in a computer system for generating content, and the decoder can be integrated in a smart phone, such as a Snapdragon 820 Hexagon DSP or the like, allowing the smart phone to receive and decode from the smart phone. The content of the computer system wireless streaming. In one example, this allows the computer system to stream over a local wireless connection, but in another example, this can be used to provide a cloud-based digital reality engine from a mobile phone network or similar network. content.

An example of this hardware configuration will be described in more detail with reference to the third figure.

This example will be exemplified with a separate hardware encoder and decoder, but it should be understood that this is not required and the same technique can be used in conjunction with integrated hardware. Furthermore, although reference is made to a virtual reality application, this is not necessary, and the technique can be applied to any environment in which image data is to be transmitted, especially when image data is to be transmitted using limited bandwidth while maintaining acceptable images. Quality and latency expectations, as in virtual reality, augmented reality or telepresence applications.

In this example, device 300 again includes a processing system 310, an encoder 320, a decoder 330, and a display device 340 of the HMD or similar form. Each of these components will be described in more detail at this time.

In this example, processing system 310 includes a microprocessor 311, a memory 312, an optional input/output device 313, such as a keyboard and/or display, and a interconnect interconnected by a busbar 315 as shown. External interface 314. In this example, external interface 314 can be used to connect processing system 310 to peripheral devices such as communication networks, storage devices, peripherals, and the like. Although a single external interface 314 is shown, this is merely an example and may actually provide multiple interfaces using many methods (eg, Ethernet, serial, USB, wireless, etc.). In this particular example, the external interface includes at least one data connection, such as a USB, and a video connection, such as DisplayPort, HMDI, Thunderbolt, and the like.

In use, microprocessor 311 executes instructions stored in the form of application software in memory 312, allowing the required processing to be performed. The application software can include one or more software modules and can be executed in a suitable execution environment, such as an operating system environment.

Accordingly, it should be appreciated that processing system 310 can be formed from any suitable processing system, such as a suitable program editing PC or the like. In one particular example, processing system 310 is a standard process Systems, such as the Intel architecture processing system, execute software applications stored in non-volatile (eg, hard disk) storage devices, although this is not required. However, it should also be understood that the processing system can be any electronic processing device such as a microprocessor, a microchip processor, a logic gate configuration, a firmware selectively associated with the implementation logic, such as an FPGA (field programmable gate) Array), a dedicated integrated circuit (ASIC), a system on a chip (SoC), a graphics processing unit (GPU), digital signal processing (DSP), or any other electronic device, system, or configuration.

Still further, although the processing system 310 is shown as a single entity, it should be understood that the processing system 310 should be formed by a plurality of physical devices that can be selectively distributed at some geographic locations, such as a portion of the cloud environment.

The encoder 320 typically includes an encoder input buffer 321 coupled in sequence to an encoder processing device 322, an encoder output buffer 323, and a transceiver 324. An additional data buffer 325 can be provided for coupling to the transceiver 324.

In use, the image material, and in one particular example, the video material has been received and temporarily stored in the input buffer 321 before being sent to the encoder processing device 322 for compression. In this regard, the encoder input buffer typically buffers the next seven columns of pixels corresponding to the image, and then the image data for the next eight columns of pixels. This allows the encoder processing device 322 to obtain pixel data for the next 8x8 block pixel from the buffered image data and begin encoding.

Once it has been completed, the next eight pixels are buffered, repeating this step until the pixel data from the eight columns of pixels has been acquired and encoded. This process is then repeated for subsequent pixel columns within the image until the pixel data for the entire image has been acquired, at which point the next image is processed in a similar manner. As a result of this method, the encoder input buffer never needs to store more than seven columns and eight image data pixels, reducing the memory requirements. In addition, for the acquired pixel This can be processed immediately using this encoding process, even before buffering the next eight pixels of the image data. This significantly reduces processing time and helps reduce overall latency.

The resulting compressed image data is then stored in the encoder output buffer 323, for example, sequentially through the readings within the encoded bits, thereby being passed through the transceiver 324 for parallel alignment to the sequence bits prior to transmission to the decoder 330. Tuple encoding. Transceiver 324 is also adapted to transmit other data, such as a sensor data received from HMD 340, through encoder data buffer 325.

According to a preferred embodiment, the buffers 321, 323, 325 can be in the form of any suitable temporary storage device, according to the preferred implementation and in one example, a high performance FIFO (first in first out) field memory chip, etc. Wait. The input buffer is typically connected to an HDMI port, a display port, or any other suitable video source, and a data buffer 335 is connected to the USB port, thereby allowing an equivalent connection to the computer system.

The transceiver 324 may be any suitable form, in one example, allowing short-range radio communication between the encoder and the decoder 360, for example, through direct peer connection WiFi ^TM, 60GHz wireless technologies.

Processing device 322 can be any device capable of performing the compression process described herein. Processing device 322 can include general purpose processing devices that operate in accordance with software instructions stored in memory. However, in one example, to determine an appropriate fast compression time, the processing device includes a customized hardware configured to perform the compression process. This should include firmware that is selectively associated with the implementation logic, such as an FPGA (field programmable gate array), a graphics processing unit (GPU), and a dedicated integrated circuit (ASIC, Application-Specific Integrated). Circuit), a system on a chip (SoC), a digital signal processor (DSP), or any other electronic device, system, or configuration. Better In an example, encoder processing device 322 is arranged to perform individual channel parallel processing of each DCT, as well as parallel encoding of individual frequency coefficients. As such, although a single encoder processing device 322 is shown, in practice, a separate encoder processing device 322 can be provided to simultaneously encode each of the channels, or alternatively, a GPU or other similar parallel processing architecture can be used. In the case of a channel uncoded such as the Y channel, the encoder processing means can simply import a delay into the encoder output buffer 323 to determine that it is still associated with the encoded CbCr. Channel synchronization.

The decoder 330 typically includes a transceiver 334 coupled to a decoder input buffer 331 and then coupled to a decoder processing device 332 and a decoder output buffer 333. Additionally, a separate data buffer 335 is provided that is coupled to the transceiver 334.

In use, the compressed image material is received from encoder 320 via transceiver 334 and temporarily stored in input buffer 331 before being sent to decoder processing buffer 332 for decompression. The resulting image data is then stored in the decoder output buffer 333 prior to transmission to the display device 340. Transceiver 324 is also adapted to transmit other data, such as a sensor data received from display device 340, through decoder data buffer 335.

According to a preferred embodiment, the buffers 331, 333, 335 can be in the form of any suitable temporary storage device, and in one example, can include a high performance FIFO (first in first out) field memory chip or the like. The output buffer is typically connected to the HDMI port and the data buffer 335 is connected to the USB port, thereby allowing an equivalent connection to the display device.

The transceiver 334 may be any suitable form, in one example, allowing short-range radio communication between the encoder and the decoder 360, for example, through direct peer connection WiFi ^TM, 60GHz wireless technologies.

The processing device 332 can include a device that operates in accordance with software instructions stored in the memory. Use a processing device. However, in one example, to determine an appropriate low decompression time, the processing device includes a customized hardware configured to perform the decompression process. This should include firmware that is selectively associated with the implementation logic, such as an FPGA (field programmable gate array), a graphics processing unit (GPU), and a dedicated integrated circuit (ASIC, Application-Specific Integrated). Circuit), a system on a chip (SoC), a digital signal processor (DSP), or any other electronic device, system, or configuration. In a preferred example, decoder processing device 332 is arranged to perform individual channel parallel processing of each DCT, as well as parallel encoding of individual frequency coefficients. Likewise, while a single decoder processing device 332 is shown, in practice, individual decoder processing devices 332 can be provided to simultaneously encode each of the channels, or alternatively, a GPU or other similar parallel processing architecture can be used. In the case of a channel uncoded such as the Y channel, the decoder processing means can simply introduce a delay in the transmission of the individual data to the decoder output buffer 333 to determine that this is still associated with the decoded CbCr. Channel synchronization.

The display device 340 includes at least one microprocessor 341, a memory 342, an optional input/output device 343, such as a keyboard or input button, one or more sensors 344, a display 345, and an external interface 346. Interconnected through a bus 347 as shown.

Display device 340 can be in the form of an HMD, and thus is provided in a suitable housing that can then be worn by a user and includes an associated lens that allows viewing of the display, as will be appreciated by those skilled in the art.

In this example, the external interface 347 is adapted to normally connect the display device to the processing system 310 via a wired connection. Although a single external interface 347 is shown, this is merely an example and may actually provide multiple interfaces using many methods (eg, Ethernet, serial, USB, wireless, etc.). In this particular example, the external interface typically includes at least one data link Connected, like USB, and video connections, like DisplayPort, HMDI, Thunderbolt, and more.

In use, microprocessor 341 executes instructions stored in the form of application software in memory 342, allowing the required processing to be performed. The application software can include one or more software modules and can be executed in a suitable execution environment, such as an operating system environment. Therefore, it should be understood that the processing device can be any electronic processing device such as a microprocessor, a microchip processor, a logic gate configuration, a firmware selectively associated with the implementation logic, such as an FPGA (field programmable gate array) ), a graphics processing unit (GPU), a dedicated integrated circuit (ASIC), a system on a chip (SoC), digital signal processing (DSP), or any other electronic device, system, or configuration.

The sensor 344 is generally used to sense the orientation and/or position of the display device 340 and may include an inertial sensor, an accelerometer, or the like. Additional sensors, such as light sensors or proximity sensors, can be provided to determine if the display device is currently being worn, and an eye tracking sensor can be used to provide an indication of the user's gaze point.

In one example, the display device can thus display device is a conventional commercial, such HTC Vive ^TM, Oculus Rift ^TM Playstation VR ^TM wearing or group, it should be appreciated that this is not necessary and may use any suitable configuration.

An example of the operation of the image compression/decompression will be described in more detail.

For the purposes of this example, assume that processing system 310 is executing an application software that produces content displayed on display device 340, wherein sensor data from sensors 345 on display device 340 and other selected sensors, Dynamically display content, such as a handheld controller or position detection system (not shown), as will be appreciated by those skilled in the art.

The operations performed by processing system 310 are performed by processor 311 in accordance with instructions stored in memory 312 as application software, and/or through I/O device 313 or other peripherals. (not shown) is received from the user's input command to execute. The actions performed by display device 340 are performed by processor 341 in accordance with instructions stored in memory 342 as application software.

The encoder 320 and the decoder 340 serve as an interface between the processing system 310 and the display device 340, allowing the image data to be compressed, wirelessly transmitted, and then decompressed before being displayed on the display device 340, while also allowing the sensor data or inputting instruction data back. Passed to the processing system. The actions performed by encoder 320 and decoder 330 are typically performed by respective processing devices 322, 332 in accordance with defined program editing, and in one example, according to a customized hardware configuration and/or embedded The instructions in the firmware are executed.

However, it should be appreciated that the above-described configuration assumed for the following example purposes is not required and that many other configurations may be used, for example, the functionality of the encoder and decoder may be directly built into processing system 310 and display device 340. Inside. In addition, the compression technique can be applied to a wide variety of other situations, including compressing and decompressing images on one or more computer systems without the need for separate display devices. However, the above configuration is particularly advantageous for virtual or augmented reality applications, telepresence applications, and the like.

An example procedure for compressing and subsequently decompressing image data will be described with reference to Figures 4A through 4D.

In this example, at steps 400 and 402, encoder 320 receives image data from processing system 310, particularly video material representing a series of images, and then temporarily stores it in encoder input buffer 321 . The image data is then analyzed, for example by parsing the data to identify a flag within the data defining the header, identifying the beginning of an image, etc., at step 404, allowing access to the next 8x8 pixel block. video material. In this regard, when buffering the data, the encoder requests an initial 8x8 pixel block from the image to begin processing. therefore, Before the start of processing, the encoder input buffer 321 is filled with the first seven rows of pixels of an image, and the first eight pixels of the eighth row of pixels. As the next eight pixels are received, the next 8x8 block is processed, and then all pixels in the first eight columns of the image have been processed. The next eight column groups are then processed in a similar manner.

The image material is typically in the form of multi-channel RGB data and then converted by processing device 322 to YCbCr luminance and chrominance channels at step 406. This processing can be performed with known mathematical coordinate transformations and will therefore not be described in further detail.

At step 408, a 2D DCT is applied to each of the luminance and chrominance channels, thereby converting the channels to the frequency domain. This processing can be performed using known techniques, and in the preferred example, is performed by processing device 322 in a very high degree of parallelism, thereby reducing processing time. The result of the conversion process on each channel is an 8x8 matrix with 64 frequency coefficients representing the strength of different frequency components within the respective image channel.

At step 410, a scaling factor is applied to each matrix, for example by dividing the scaling factor by each frequency coefficient, thereby reducing the strength of each frequency coefficient. The scaling factor should be any value and can be different for different channels. For this part of the process, the scaled frequency coefficients are usually reduced to a set number of integers or significant pictures, thereby reducing the amount of data required to encode the frequency coefficients, for example, the coefficient 500 should require 9 bits. To encode, and 50 only requires 6 bits, so applying a scaling factor of 10 reduces the number of bits needed to encode the particular frequency coefficients.

In this way, it should be understood that the lower value coefficient can be reduced to zero, for example 4 is scaled to 0.4 and thus rounded to zero. Although this leads to loss of information, this tends to occur at higher frequency components that are known to contribute less to the overall appearance of the image, so the loss of this loss Hit limited.

After applying the scaling factor, the selection rule is determined at step 412. The selection rules are used to allow selection of certain of the frequency components such that other components may be discarded at step 414. The selection is typically performed according to a hierarchy defined as diagonal across the hierarchy of the matrix, such an 8x8 frequency domain matrix comprising 1, 2, 3, 4, 5, 6, 7, 8, 7, 6 respectively. 15 levels of 5, 4, 3, 2, 1 coefficients.

The levels that have been discarded are usually higher frequency components, and there is a mention of a lower contribution to the appearance of the image. According to the preferred implementation, the selection rules define which level is discarded based on a wide range of factors.

For example, the chroma channel's contribution to the appearance of the image is usually lower than the brightness channel, so more levels of chromaticity are usually discarded relative to the brightness channel. Thus, for example, for the luminance channel Y, levels 1 through 8 are reserved, corresponding to 36 frequency coefficients, as shown in 531 of the fifth diagram, and levels 1 through 6 may be reserved for the chrominance channels Cb and Cr, Corresponds to only 21 frequency coefficients, as shown in 532 and 533 in the fifth figure. It will be appreciated that this reduces the total amount of frequency coefficients required to encode all of the three channels from 192 to 78.

Which hierarchy to keep is selected according to the rules, and when selecting which hierarchy to keep, the encoder and decoder are allowed to apply the same criteria, and then executed in an instant adaptation manner. Then select different levels according to other factors, such as the quality and/or bandwidth of a communication channel between the encoder and the decoder. Thus, if the bandwidth is reduced due to interference, fewer levels can be selected, which results in a higher degree of compression. Although this will result in a decline in image quality, in many applications, such as VR, some frame quality degradation is less noticeable than dropping the frame or increasing the delay, so it is the better result.

Other factors may include, but are not limited to, movement of the display device, especially movement Rate, image display requirement, target display resolution, position of the pixel array within the one or more images, and observation of the element array relative to the one or more images within the one or more images The gaze is at the position of the point. Further examples are explained in more detail below.

At step 416, a one-bit encoding scheme is determined, and then at step 418, the selected frequency coefficients are selectively encoded using different number of bits. In this scheme, as described above, a scaled frequency component may only require 6 bits to fully encode the coefficient. Therefore, the bit coding scheme is chosen such that for each level in the hierarchy, the number of bits used to encode the frequency coefficients is changed. This is possible because higher frequency components typically have less intensity and therefore require fewer bits for encoding.

In one example, the encoding scheme for the luma channel is different from that used for the chroma channel, and the example encoding scheme is shown in Table 1 below.

An example of this coding form for the first six levels of each of the three channels is shown in the fifth figure. This particular combination of selection rules and bit encoding schemes results in a luminance channel 541 encoded in 129 bits, and each of the chroma channels 542, 543 is encoded in 60 bits, resulting in encoding the 8x8 using only 249 bits. All three channels of the pixel array. It should be understood that this is compared to The original uncompressed image data of 192 bytes is required, representing more than six times the compression.

From this it will be appreciated that an encrypted channel having 0 bits effectively corresponds to discarding the channel, so this can be used as a way of selecting the encoded coefficients by selecting a coding scheme using the encoding rules, thereby effectively Steps 412 and 414 are combined with 416 and 418.

Once the encoding has been performed, the parallel encoded bit components can be concatenated into a one-bit stream by performing parallel-to-sequence byte encoding at step 420, and then 32-bit tuples (256 bits) are allowed. ) said. At step 422, the octets are parsed to identify subsequent contiguous contigs at step 424. In particular, this approach is used to identify subsequences of three or more consistent bytes and then replace it with a code at step 426 without losing any information.

In particular for most images, there is a zero string within the resulting coded frequency coefficient, where the scaled coefficients are all reduced to zero. Thus, a code can be substituted, which can be recognized by the decoder, allowing the decoder to re-insert a subsequence of a consistent byte.

Although the code can be in any suitable form, in one example, the code includes a header identifying the particular byte as a code, and information corresponding to the value and number of the consistent byte. In a preferred configuration, a 2-byte code is combined with a column (1-8) zero number using a Boolean OR operation. In one example, the zero number is represented as N-1, and the number of 0-7 is ORed with the 2-byte code, so only the third bit of the second byte is used, for example: used The code can be (1111 1111; 1111 1000), where the second byte is ORed according to the zero number and 0-7. It should be appreciated that a similar approach can be used for different values.

This method works well because the code rarely causes the continuous number to exceed or equal to 248, so the decoding algorithm can simply search for a byte with a value of 255 and a subsequent byte with a value greater than or equal to 248, It is identified as being opposite to the encoded frequency component code. This code is then replaced with a byte corresponding to the data having a series of zeros represented by the last 3 bits of the second bit. This can result in a further reduction of 19-25% of the data after the bit coding phase according to the latest test.

After the code replacement is performed, the compressed image data is output at step 428. In particular, the compressed image data is typically stored in output buffer 323 until sufficient data is available, at which point a data packet is created and transmitted by transceiver 324 to the encoder.

At step 430, decoder 330 receives the compressed data via transceiver 334 and stores it in decoder input buffer 331. The data is parsed at step 432 to identify the code within the data, as described above, and then replaced with a subsequence of repeated consistent bytes at step 434, thereby reestablishing the bit of the bit encoded frequency coefficient. Streaming.

At step 436, decoder processing device 332 determines the bit encoding scheme used to encode the image material, and uses this scheme to perform the sequence of the bit stream to parallel decoding, thereby determining at step 438 that the Coding frequency coefficient. In particular, this allows the decoder processing means 332 to determine the number of bits used to encode the frequency coefficients, allowing the bit stream to be effectively divided into individual coded frequency coefficients. The scaled frequency matrices are re-established using the individual frequency coefficients at step 440, the discarded frequency coefficients are generated using null values, thereby filling the entire matrix, and then scaled according to the scaled down coefficients at step 442.

At step 444, reverse 2D DCT conversion is applied prior to converting the converted matrix of each YCbCr channel to an RGB channel at step 446, allowing an 8x8 pixel block to be output at step 448, allowing execution by display device 340. .

Therefore, the above processing can significantly reduce the amount of image data required for each 8x8 pixel block encoding, thus reducing the overall image amount. In particular, 2D DCT using YCbCr channels, pair The result is that the frequency coefficient is selectively bit encoded and an optional final non-destructive coding scheme is achieved. Each channel can be processed simultaneously, applying the DCT to the 8x8 array in parallel. Furthermore, the bit coding of each of these frequency coefficients also occurs simultaneously, producing a large number of parallel methods that allow simultaneous compression and decompression to be performed in a rapid manner, with minimal impact on latency, such as VR, AR, and telemetry. This type of instant application is quite important.

An example of the way in which the selection rules are implemented will be described with reference to the sixth and seventh figures.

In particular, this example focuses on how to use the spatial localization of the pixel array within an image to affect the selection process and thus the amount of compression performed. Similar processing can be performed for other factors and will not be described in detail herein. This process is the same for the encoder or decoder, so for ease of illustration, only focus is placed on the operation of the encoder.

In this example, at step 600, encoder processing device 322 determines the selection rules for the current channel. In this regard, as previously described, different rules are typically applied to each of the luminance and chrominance channels. Then at step 605, encoder processing device 322 determines the location of the array of pixels within the image and thereby determines if an absolute rule associated with the location is accompanied. In this regard, when an image is displayed using a headset, the lens configuration generally indicates that the portion of the image is not visible. This example is shown in the seventh diagram, which shows a viewable portion 701 of an image and an unviewable portion 702. In this regard, if the pixel array is located within the unviewable portion 702, the pixel array need not be encoded, so the process can simply ignore all levels of the frequency coefficients. Thus, if there is a rule for the absolute position of the pixel array at step 610, then the process proceeds to step 625, allowing for individual level selection to be determined, and then a separate bit coding scheme is used at step 630, as described above with respect to steps 414 through 418. instruction of.

Otherwise, at step 615, encoder processing device 322 determines the observer gaze point for the image. It can be achieved by using an eye tracking sensing system, which can be understood by a skilled person. Once determined, a relative position is determined at step 620, and the distance between the gaze point and the current pixel array position in an example is determined, and then used at steps 625 and 630 to direct the coefficients and/or bit coding. Program decision making. In this respect, when a person's peripheral field of view does not feel the same level of detail as his focus position, a greater degree of compression can be further used from the user's gaze point.

In one example, the user's gaze point can be defined as the lowest compression area, and the degree of compression is increased as moving outward from the lowest area. The area can be any shape and can be circular, elliptical, oval, etc. depending on the particular environment. This area can also be off center with respect to the gaze area, thus ensuring that a minimum compression below the gaze point is used, which is generally the area that the user will perceive.

It should also be understood that this is predictable in an environment where the user's gaze point cannot be measured. In general, the prediction is focused on the center of the picture, but it should be understood that this is not required and can be changed, for example, depending on the nature of the content represented, the direction in which the headset is moved, and the like.

As such, it should be appreciated that this illustrates a mechanism that allows for selective encoding to be performed based on the absolute and/or relative position of the pixel array within the image. Having the encoder and decoder each perform the same algorithm to select the encoding scheme and the selection used, which allows the decoder to reliably and accurately decompress the compressed image data while still allowing dynamic encoding to be performed, Maximize overall compression while minimizing perceived image quality loss.

It should be appreciated that similar approaches can be used for other factors, such as transmission quality and/or bandwidth, so it is determined that this degree of compression is best suited to the current environment.

Further examples of methods for compressing image data and subsequently decompressing the compressed image data will be described with reference to the eighth figure.

Although illustrated by a separate example, it will be appreciated from the following description that this current example can be used in conjunction with the compression/decompression techniques described above to further improve compression and computational efficiency. In particular, the above technique represents one of the options available in the following processing, but it is also understood that other compression techniques may be appropriate depending on the environment.

In this example, at step 800, pixel data representing a pixel array within the one or more images is obtained from the pixel data. This can be accomplished in any suitable manner, but typically involves buffering the received image data until a pixel array, such as an 8x8 array, has been obtained in a manner similar to that described above.

At step 810, a coding scheme has been determined, typically selecting one of the one or more previously defined coding schemes, such as the coding scheme described above with respect to the first to seventh figures. The selection may be performed in any suitable manner, such as by analysis of the image and/or pixel data, by image type associated with the image, by instructions from the image source or display, etc., and this is typically based on current conditions Requirements, execution to provide higher or lower compression.

At step 820, the pixel data is encoded using the encoding scheme, such as using the techniques described above, and the like. At step 830, an encoded code indicating the encoding scheme is, for example, by a look-up associated with the available scheme, which is used in conjunction with the encoded block to generate compressed image material. In one example, this may be accomplished by adding the encoded code as a prefix code to the encoded pixel data, although other methods may be used, including simply replacing the pixel with the encoded code depending on the encoding scheme used. Data, for example, is replaced with a code character.

Once the compressed image material has been created, this can then be transmitted to a destination, such as a portion of a tuple stream, allowing decoding to produce image material.

At step 840, this processing is typically performed by determining the encoded code from the compressed image material. In this regard, the encoded code typically has a set of formats, such as a particular combination of bits, that is readily identifiable within the received byte stream. The encoding code is then used to determine the encoding scheme to use, typically through the lookup procedure at step 850. Then, at step 860, the encoded image data is decoded using the recognized encoding scheme, for example, using the manner described above with respect to the first to seventh figures, thereby generating a representation of the one or more images. Pixel data of the pixel array.

Therefore, the above processing allows encoding to be performed using one of some encoding schemes. In a preferred example, it is known in advance that the encoder and decoder use the different encoding schemes to allow for dynamic changes to each pixel array within the encoded/decoded image. This determines that the encoding scheme is used to optimize the particular material being encoded, thereby maximizing compression while determining to maintain other desired characteristics, such as image quality and encoding time.

Some further features will be described at this time.

When encoding the image, the process typically involves one or more of a range of factors, including the image type of the image data, an encoding scheme indication received from an image data source or display, and utilizing at least the image. The data and one of the pixel data, or according to compression requirements, such as the required amount of compression, the resulting image quality, compression delay, etc., determine the coding scheme.

Thus, for example, when encoding photos rather than computer generated graphics, different encoding types can be used because they typically have very different properties. From the relay data associated with the image, Or depending on the properties of the image itself, the nature of the image can be obtained. In addition, an image source that supplies the image data, such as a computer system that supplies the image data for transmission to a display or other computing device, can specify the encoding scheme that should be used.

A further example is to analyze the pixel data to identify characteristics of the pixel array. For example, if the pixel array has a single color, then a different algorithm than that used for the pixel array containing multiple colors should be used. This is especially useful for encoding large-area images with a single color, such as sky or background, allowing a single encoding code that indicates the use of a solid color to replace the pixel data of the entire pixel array, thereby resulting in maximum compression without effective image quality. loss.

Further examples are the identification of gradients or boundaries, which can result in the use of many image compression methods or the desired compression, and thus alternative compression methods may be required.

In a preferred example, wherein the pixel array is not a solid color, the manner described with respect to the first to seventh figures is used. In the event that the pixel array does not contain gradients or boundaries, the technique may involve encoding all three of the YCbCr channels. However, in the event that the pixel array contains a gradient or boundary, the Y channel can be encoded without the need to retain additional luminance information. Although this reduces the total amount of final compression obtained, the total amount of compression is still sufficient due to the compression of the CbCr channels, and the additional retained color information can significantly improve the quality of the resulting image in the gradient or boundary region.

This selection can also be used to control other aspects of a compression algorithm, for example: this applies to specifying which frequency coefficient to retain in the compression process described above, similar to the manner in which the process is performed in the sixth figure.

The coded code is typically specified as a digital form from 0 to 255, which can be defined with a single byte and allows for up to 256 different coding schemes to choose from. So wide Encoding schemes can be used, and these schemes are each assigned to the numbers 0 to 255, so that a wide variety of encoding schemes can be used, thereby achieving optimal compression with the desired image quality. In a specific example, it should be understood that an industry standard list of coding schemes can be established so that the coding and decoding system can operate in concert even if the coding scheme to be used is not communicated before.

In a particular example, the coded code may be identified by a fixed format preamble byte such as "11111111" such that the code code used to select the coding scheme assigned to the number 1 should be "1111111100000001". It should be appreciated that performing this allows the decoding process to easily identify the encoding process used. However, any suitable means can be used and the above examples are not intended to be limiting.

It will be appreciated that when decoding the compressed image material, a similar approach is used, which involves replacing an encoded code with a solid color pixel array, or using the previously described decoding process.

It should also be appreciated that the image compression process can be performed using a device similar to that described above and therefore will not be described in further detail.

At this time, referring to the ninth A map and the ninth B graph, further example processing will be described in more detail.

In this example, at step 904, encoder 320 receives image data from processing system 310, particularly video material representing a series of images, and is then temporarily stored in encoder input buffer 321 at step 902. The image data is then analyzed, for example, by parsing the data to identify a flag within the data defining the header, identifying the beginning of an image, and the like, and acquiring image data corresponding to the next 8x8 pixel block at step 904. It should be understood that this corresponds to steps 400 through 404 above and will therefore not be described in further detail.

At step 906, encoder 920 selects an encoding scheme to use. This can be seen The analysis of the 8x8 pixel block, for example, determines whether the block is a single color, gradient, edge, etc., or can be implemented based on information provided by the image source.

In the example shown, the encoding scheme includes a first encoding scheme for performing a block replacement at step 908 in which a monochrome pixel block is replaced with a known predetermined code. A second encoding scheme is provided at step 908, which in this example corresponds to the encoding scheme described above, and is assigned to steps 406 through 426. However, it should be appreciated that any number N of different schemes can be used, and then other schemes are selected and used at step 910.

At step 912, an encoded code representing the selected encoding scheme is prepended to the encoded material, and then the data is output as compressed image data at step 914.

Next, in step 916, the decoder 330 receives the compressed image data to perform the decoding, and in step 918, detects the presence or absence of the encoded code, and then, at step 920, uses the detection result to select a Proper decoding scheme. For example, this may include performing a block replacement at step 922, replacing the predetermined code with a monochrome pixel block, or including performing the decoding process of steps 432-446 at step 924. Similarly, appropriate decoding scheme to provide N, and other programs can be selected and used in step 926 in accordance with requirements.

Finally, at step 928, an 8x8 pixel block is output from the decoder.

From this, it will be appreciated that the selection procedure executed at step 906 performs the classification function to be performed such that each pixel block can be assigned to a different coding scheme. In one example, up to 256 different schemes can be defined, allowing a third party to import a new compression/decompression scheme to the 8x8 pixel block within the previously illustrated architecture. Further, the user can define a specific classification scheme, such as, for example, according to the nature of the image content to be encoded, the needs of the situation, such as delay requirements, etc. For example, different combinations of 256 schemes can be used in different environments.

This greatly increases the flexibility of the executable encoding, and in particular allows the use of classification processing for each pixel block to dynamically select individual encoding schemes, thereby optimizing the encoding of each chunk as the case may be. Still further, different classification schemes can be provided, each of which allows access to different combinations of coding schemes, thereby further increasing flexibility, allowing the user to ensure that optimal coding can be performed in a variety of different situations.

In another broad form, the mobile compression scheme described in the sixth and seventh figures above may also be used in conjunction with different encryption schemes to provide an image forming portion from one or more of the digital reality content. The method of compressing image data will be explained with reference to the tenth figure.

In this example, the method includes obtaining pixel data from the image data at step 1000, the pixel data representing an array of pixels within the one or more images. This can be done in a manner similar to that described above.

At step 1010, a position of the pixel array within the one or more images relative to a defined location is determined, and the defined location at least partially indicates a gaze point of the user. The defined location may be based on the actual measurement point of the gaze, or the expected or predicted gaze point of the user, for example by assuming that the user is beginning to stare at the approximate center of the image, or based on content, such as focus within the image, wearing The group moves and so on. In addition, the defined point may deviate from the gaze point, such as positioning it below the gaze point, to take into account that the individual tends to focus slightly below the gaze point to avoid hitting the obstacle while walking.

Then, the pixel data is compressed at step 1020 to generate compressed image data, and the pixel data is compressed at least partially according to the determined position, so that the degree of compression is taken Depends on the determined position of the pixel array.

It will be appreciated that a similar decompression process can be performed which involves obtaining compressed image data at step 1030, the compressed image data representing an array of pixels within the one or more images, and based at least in part on such The pixel array within one or more images is compressed relative to a position of a defined location that at least partially indicates the gaze point of the user, and at least partially in accordance with the determined position at step 1040 Uncompress the compressed image data.

As such, this provides a mechanism for compressing and subsequently decompressing the image, wherein the compression is controlled based on the position of a pixel array relative to a defined point, and then controlled based at least in part on the predicted or measured gaze point of the user. In particular, this allows the degree of compression to be selected based on the position of the array of pixels such that less compression can be used in the vicinity of the gaze point, while greater compression can be used in areas remote from the gaze point, such as within the perimeter of the field of view of the user. This actually provides foveated compression by increasing the compression of the user's peripheral field of view, where the reduced image quality is less noticeable, allowing for greater overall compression without noticeable quality loss.

It will be appreciated that this process can be used in conjunction with the compression process described above, for example to allow selection of different compression schemes based on the location of the pixel array and to allow for the use of the above described bit coding scheme. However, this is not required and any suitable compression scheme can be used, such as wavelet compression, adaptive sampling, and the like.

In any event, some further features will be described at this time.

In one example, the defined location is the user's gaze measurement point, the user's gaze expected point, the deviation from the user's gaze measurement point, the deviation from the user's gaze at the expected point, and at least in part in accordance with the user's gaze point. Staring at one of the points determined by the data, wherein the condensation The information was obtained from a gaze tracking system. As such, the defined points may be based on the measured or predicted points of the gaze and may be offset, such as below the gaze point, to optimize the degree of compression without causing a noticeable degradation in image quality.

The degree of compression may be based on a distance from the defined point, such as gradually decreasing away from the point of gaze, but may also be based on a direction relative to the defined point, such that the compression is greater when the point is above or below the defined point. It will be appreciated that this allows a corresponding degree of compression to be used in positioning an area having an arbitrary shape relative to the gaze point, and this may be configured according to the particular environment and/or nature of the compressed content. For example, this allows an elliptical, oval or heart-shaped region around the defined point to have a reduced amount of compression compared to the surrounding area, thereby optimizing image quality in areas where the user's perception of any compression will be greater. .

In one example, the method includes selecting one of a plurality of coding schemes and encoding the pixel data using the selected coding scheme, such that it is understood that this allows for combining techniques similar to those described in the eighth and ninth diagrams To perform rendering compression, the encoding scheme used is selected according to the location of the pixel array. Thus in one example, each coding scheme provides a corresponding degree of compression, and wherein the method includes selecting an encoding scheme based at least in part on the desired degree of compression and/or the location of the array of pixels.

The process also involves deciding an encoding code indicating the encoding scheme used and using the encoding code and the encoded pixel data to generate compressed image data, thereby allowing it to be used to decode the compressed image material. However, this is not required, or the decompression process may involve the use of similar criteria to determine the coding scheme used, such as the location of the pixel array and thus the degree of compression required.

In one example, the encoding scheme is similar to that described above with respect to the first graph. In the present case, by applying a conversion to the pixel data, a set of frequency coefficients indicating the frequency components of the pixel array is determined to compress the image data, and at least some of these are selectively encoded using a one-bit encoding scheme. A frequency coefficient that produces a set of encoded frequency coefficients and uses the encoded frequency coefficients to produce compressed image data. However, it should be understood that any suitable compression technique, such as JPEG, etc., can be used.

In the preferred embodiment, as in the case of the first figure above, the bit coding scheme defines the number of bits used to encode each of the frequency coefficients, such that at least some of the encoded frequency coefficients have different bits. Yuan. More typically, a smaller number of bits are used to encode frequency coefficients corresponding to higher frequencies, and preferably, a progressively smaller number of bits are used to encode frequency coefficients corresponding to increasing frequencies. In one example, at least some of the frequency coefficients (effectively encoded using zero bits) are discarded such that the set of encoded frequency coefficients is less than the set of frequency coefficients, and these frequency coefficients generally correspond to higher frequencies.

In one example, the method includes selecting one of a plurality of bit coding schemes and encoding the frequency coefficients in accordance with the selected bit coding scheme. It should be understood that different coding schemes may correspond to the coding scheme of the first figure, in which different bit coding schemes are used to encode different frequency coefficients with different number of bits to provide different degrees of compression. .

It should thus be appreciated that the bit encoding scheme can be selected based at least in part on the degree of compression desired and/or the location of the array of pixels. This may be based on other factors, such as the location of the pixel array, the transmission bandwidth of a communication link for transmitting the compressed image data, the transmission service quality of a communication link for transmitting the compressed image data, and a display device. The movement, image display requirements, a target display resolution, the channel being processed, the error matrix, and so on. As above, This allows the compression to be dynamically adjusted to help optimize the compression and the ability to obtain the best possible image quality in the current situation.

In one example, the frequency components are arranged in a plurality of levels, and wherein each bit encoding scheme defines a number of different bits to be used to encode the frequency coefficients in each of the plurality of levels. In particular, where the array is an N x N pixel array, this produces a frequency component of 2 N -1 order, with components within each level being encoded with individual bit numbers.

As mentioned above, the method can include applying a scaling factor in a manner similar to that described above.

The image data typically defines a plurality of channels, wherein the method includes selectively encoding frequency coefficients for each channel. For example, the pixel data can define RGB channels that are converted to YCbCr channels and then converted as described above. The process involves selectively encoding more frequency coefficients to the Y channel than the Cb or Cr channels, selectively encoding frequency coefficients to the YCbCr channels and/or selectively encoding frequency coefficients to the CbCr channels and using the Y Channel, so the Y channel is effectively uncompressed.

In one example, the conversion is a 2-D discrete cosine transform.

The method also generally includes buffering the image data to obtain the array of pixels, such as by n-1 columns of buffered pixels, and subsequent n pixels of the next column of pixels to form a first n x n pixel block. The contiguous blocks are then buffered as previously described, wherein the number of pixels in each block can be selectively controlled based on the selected bit coding scheme, the desired degree of compression, and/or the location of the pixel array.

The frequency coefficients are typically encoded simultaneously, and the compressed image data is generated by juxtaposition to the sequence byte encoding, although other suitable methods may be used.

A similar approach can be used to decode the compressed image material.

For example, decoding the compressed image data can include selecting one of a plurality of decoding schemes and decoding the pixel data using the selected decoding scheme, wherein the selection is based on a desired degree of compression, a location of the pixel array, or the An encoded code in the compressed image data.

The method can include using a decompression scheme similar to that described above, which can use one of a plurality of bit coding schemes to provide varying degrees of compression by encoding individual bits for the frequency coefficients in each of the plurality of levels Yuan.

A scaling factor can be applied to at least some of the frequency coefficients, such that the scaled encoded frequency coefficients are decoded, the scaling factor used to increase the strength of each frequency coefficient. Again, different scaling factors can be applied to at least some of the encoded frequency coefficients, or the same scaling factor can be applied to each encoded frequency coefficient. In both cases, different scaling factors can be applied to the encoded frequency coefficients in different channels.

A plurality of channels can be used, each of which is decoded as needed, which can be selectively performed simultaneously by using sequence to parallel byte decoding and selectively decoding frequency coefficients simultaneously.

The degree of compression desired by the above technique may be based on the location of the pixel array, the transmission bandwidth of a communication link for transmitting the compressed image data, and the transmission service quality of a communication link for transmitting the compressed image data, Any one or more of the movement of the display device, the image display request, a target display resolution, the channel being processed, or the error matrix.

These techniques can be used to transfer digital reality content, including any one or more of augmented reality, virtual reality, mixed reality, telepresence, and the like. This may include transmitting the image data from the computing device to the wearable digital reality headset via a communication network and/or a wireless communication link.

Thus, the above process allows for significant image compression without adversely affecting latency, allowing the technology to be used to provide wireless connectivity to virtual or augmented reality wearable displays using existing hardware, while also allowing other Improved image compression is more widely implemented in applications.

In the present specification and the following claims, the word "comprise" and variations such as "comprises" or "comprising" are to be understood to include the integer group or step, unless the context requires otherwise. Any other integer or integer group is not excluded.

Those skilled in the art will understand that many changes and modifications will become apparent. All such changes and modifications that the skilled artisan is aware of should fall within the broad spirit and scope of the invention described above.

Steps 100-170‧‧

Claims (122)

A method of compressing image data in one or more images forming part of a digital reality content, the method comprising: a) obtaining pixel data from the image data, the pixel material data representing the one or more images a pixel array; b) determining a position of the pixel array relative to a defined position within the one or more images, the defined position indicating at least a portion of the gaze point of the user; and c) the pixel Data compression produces compressed image data that is compressed at least in part in accordance with the determined position, whereby a degree of compression depends on the determined position of the pixel array. The method of claim 1, wherein the defined position is at least one of: a) a measured user gaze point; b) an expected user gaze point; c) an offset one Measured at the user's gaze point; d) offset from an expected point of view of the user; and e) at least in part determined by the gaze data indicative of the user's gaze point, the gaze data from Gaze through the tracking system to get it. The method of claim 1 or 2, wherein the method comprises compressing the pixel data, such that the degree of compression depends on one of: a) a distance from the defined point; b) based on a direction of the defined point; c) further increase from the defined point; and d) provide rendering compression. The method of any one of claims 1 to 3, wherein the method comprises: a) selecting one of a plurality of coding schemes; and b) encoding the pixel data using the selected coding scheme. The method of claim 4, wherein each of the encoding schemes provides a degree of compression, and wherein the method comprises selecting the encoding scheme based at least in part on at least one of: a) a desired compression Degree; and b) the position of the pixel array. The method of claim 4, wherein the method comprises: a) determining an encoding code indicating the encoding scheme used; and b) using the encoding code and the encoded pixel data to generate compressed image data. The method of any one of claims 1 to 6, wherein the method comprises compressing the pixel data by using a coding scheme: a) applying a conversion to the pixel data to determine a frequency indicative of the pixel array a set of frequency coefficients of the component; b) selectively encoding at least some of the frequency coefficients using a one-bit encoding scheme, thereby generating a set of encoded frequency coefficients; and c) generating compression using the encoded frequency coefficients video material. The method of claim 7, wherein the bit coding scheme defines a number of bits for encoding each of the frequency coefficients, and wherein the frequency coefficients are selectively encoded a code, such as at least one of: a) at least some of the encoded frequency coefficients have different number of bits; b) fewer bits are used to encode frequency coefficients corresponding to higher frequencies; c) gradually less The number of bits is used to encode a frequency coefficient corresponding to a gradually higher frequency; d) discarding at least one frequency coefficient such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and e) discarding at least corresponding to a higher frequency A frequency coefficient. The method of claim 7 or 8, wherein the method comprises: a) selecting one of a plurality of bit coding schemes; and b) encoding the frequency coefficients in accordance with the selected bit coding scheme. The method of claim 9, wherein each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different numbers of bits to provide different degrees of compression. The method of claim 9 or 10, wherein the bit encoding scheme is selected based at least in part on at least one of: a) a desired degree of compression; and b) the location of the pixel array. The method of any one of clauses 9 to 11, wherein the frequency components are arranged in a plurality of levels, and wherein each of the bit coding scheme definitions is to be used to encode each of the plurality of levels The number of individual bits of the frequency coefficients. The method of claim 12, wherein the array is an N x N pixel array, resulting in a 2N-1 frequency component level. The method of any one of claims 7 to 13, wherein the method comprises applying a scaling factor to at least some of the frequency coefficients, such that the scaled frequency coefficients are encoded, the scaling factor being used to reduce each An intensity of a frequency coefficient, and wherein at least one of: a) applies a different scaling factor to at least some of the frequency coefficients; b) applies the same scaling factor to each frequency coefficient; and c) Different scaling factors apply to the frequency coefficients in a different channel. The method of any one of claims 7 to 14, wherein the image data defines a plurality of channels, and wherein the method comprises selectively encoding frequency coefficients for each channel. The method of claim 15, wherein the pixel data defines an RGB channel, and wherein the method comprises: a) converting the RGB channel to a YCbCr channel; and b) converting the YCbCr channels. The method of claim 16, wherein the method comprises at least one of: a) selectively encoding more frequency coefficients for the Y channel than the Cb or Cr channels; b) selectively targeting the YCbCr channels Simultaneously encoding frequency coefficients; and c) selectively encoding frequency coefficients for the CbCr channels and using the Y channel. The method of any one of claims 1 to 17, wherein the conversion is a two-dimensional (2-D) discrete cosine transform. The method of any one of claims 1 to 18, wherein the method comprises: obtaining pixel data from the image data by: a) buffering image data corresponding to a next n-1 column of pixels of the image; b) buffering The next n pixels of image data of the next column of pixels; c) obtaining pixel data of the next n x n pixel block from the buffered image data; d) repeating steps b) and c) until all Pixel data is obtained in n columns of pixels; and e) steps a) through d) are repeated until pixel data has been obtained from each column of pixels of the image. The method of claim 19, wherein n is selected according to at least one of: a) a selected bit coding scheme; b) a desired degree of compression; and c) the location of the pixel array. The method of any one of claims 1 to 20, wherein the method comprises: a) simultaneously selecting a coding frequency coefficient; and b) generating compressed image data at least in part by parallel to sequential byte encoding. A method of decompressing compressed image data in one or more images forming part of a digital reality content, the method comprising: a) obtaining compressed image data, the compressed image data representing the one or more An array of pixels within the image and compressed at least in part based on a position of the pixel array relative to a defined location within the one or more images, the defined location indicating at least a portion of the gaze point of the user; b) decompressing the compressed image data based at least in part on the determined position. The method of claim 22, wherein the defined location is at least one of the following: a) a measured user gaze point; b) an expected user gaze point; c) offset from the measured user gaze point; d) offsetting the expected user gaze Where; and e) at least in part determined by the gaze data indicative of the user's gaze point, the gaze data is obtained from a gaze tracking system. The method of claim 22, wherein the method comprises: a) selecting one of a plurality of decoding schemes; and b) decoding the pixel data using the selected decoding scheme. The method of claim 24, wherein the method comprises selecting the decoding scheme based at least in part on at least one of: a) a desired degree of compression; b) a location of the pixel array; and c) indicating that the One of the encoding schemes encodes a code that is determined by the compressed data image. The method of any one of claims 22 to 25, wherein the method comprises decompressing the compressed image data using a decoding scheme: a) using the bit within each encoded frequency coefficient as defined therein a one-bit coding scheme of a plurality of elements, determining a set of encoded frequency coefficients from the compressed image data; b) performing bit decoding of the encoded frequency coefficients according to the bit coding scheme, thereby generating a set of frequency coefficients , wherein at least one frequency coefficient is generated such that the set of encoded frequency coefficients is less than the set of frequency coefficients; c) applying a reverse transform to the set of frequency coefficients to determine pixel data representing a pixel array within the one or more images. The method of claim 26, wherein the bit coding scheme defines the number of bits used to encode each of the frequency coefficients, the bit coding scheme uses a smaller number of bits to encode the corresponding a higher frequency frequency coefficient, and wherein the method includes generating at least some of the frequency coefficients corresponding to the higher frequency. The method of claim 26, wherein the method comprises: a) selecting one of a plurality of bit coding schemes; and b) decoding the encoded frequency coefficients in accordance with the selected bit coding scheme. The method of claim 28, wherein each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different number of bits to provide a different degree of compression. The method of claim 28 or 29, wherein the bit coding scheme is selected based at least in part on at least one of: a) an encoding code; b) the bit encoding scheme for generating the compressed image Data; and c) the location of the pixel array. The method of any one of claims 28 to 30, wherein the frequency components are arranged in a plurality of levels, and wherein each of the bit coding scheme definitions is to be used to encode each of the plurality of levels The number of individual bits of the frequency coefficients. The method of claim 31, wherein the array is an N x N pixel array, resulting in a 2N-1 frequency component level. The method of any one of claims 26 to 32, wherein the method comprises applying a scaling factor to at least some of the frequency coefficients, such that the scaled frequency coefficients are decoded, the scaling factor being used to increase each An intensity of a frequency coefficient, and wherein at least one of: a) applies a different scaling factor to at least some of the encoded frequency coefficients; b) applies the same scaling factor to each encoded frequency coefficient; c) Apply a different scaling factor to the encoded frequency coefficients in a different channel. The method of any one of claims 26 to 33, wherein the image material defines a plurality of channels, and wherein the method comprises selectively decoding the encoded frequency coefficients for each channel. The method of claim 34, wherein the compressed pixel data defines a YCbCr channel, and wherein the method comprises: a) performing an inverse conversion of the YCbCr channels; and b) converting the converted YCbCr channels into RGB channel. The method of claim 35, wherein the method comprises at least one of: a) generating more frequency coefficients to the Cb or Cr channel than the Y channel; b) simultaneously decoding the encoded YCbCr channels; and c Decoding the CbCr channels and converting the encoded CbCr channels to the Y channels into RGB channels. The method of any one of claims 26 to 36, wherein the inverse transformation is an inverse two-dimensional (2-D) discrete cosine transformation. The method of any one of claims 26 to 37, wherein the method comprises: a) decoding the compressed image data at least in part by sequence to parallel byte decoding; and b) simultaneously selectively decoding the frequency coefficients. The method of any one of claims 5, 11, 20, 25, and 30, wherein the desired degree of compression is determined according to at least one of: a) the location of the pixel array; b) for transmitting the The transmission bandwidth of a communication link of the compressed image data; c) the transmission service quality of a communication link for transmitting the compressed image data; d) the movement of a display device; e) the image display requirement; and f) a target display Resolution; g) processed one channel; and h) error matrix. The method of any one of claims 1 to 39, wherein the digital reality is at least one of: a) augmented reality; b) virtual reality; c) mixed reality; and d) telepresence. The method of any one of claims 1 to 40, wherein the method is for transmitting the image data from a computing device to a wearable digital device by using at least one of the following The wearable group: a) a communication network; and b) a wireless communication link. An apparatus for compressing image data in one or more images forming part of a digital reality content, the apparatus comprising at least one electronic encoder processing device, the at least one electronic encoder processing device: a) obtaining pixels from the image data Data, the pixel material data represents one pixel array in the one or more images; b) determining a position of the pixel array within the one or more images relative to a defined position, the defined position being at least a portion indicating a gaze point of the user; and c) compressing the pixel data at least in part in accordance with the determined position, such that the degree of compression is dependent on the determined position of the pixel array. An apparatus for decompressing compressed image data in one or more images forming part of a digital reality content, the apparatus comprising at least one electronic decoder processing device, the at least one electronic decoder processing device: a) obtaining compressed Image data representing an array of pixels within the one or more images and compressed at least in part based on a position of the pixel array relative to a defined location within the one or more images And the defined location at least partially indicates the gaze point of the user; and b) decompressing the compressed image data based at least in part on the determined location. A method of compressing image data representing one or more images, the method comprising: a) obtaining pixel data from the image data, the pixel material data representing the one or more a pixel array within the image; b) applying a conversion to the pixel data to determine a set of frequency coefficients indicative of a frequency component of the pixel array; c) selectively encoding at least some of the frequency coefficients using a one-bit encoding scheme, Thereby generating a set of encoded frequency coefficients, wherein the bit coding scheme defines the number of bits used to encode each of the frequency coefficients, such that when the frequency coefficients are selectively encoded: i) different uses The number of bits encodes at least some of the encoded frequency coefficients; and ii) discards at least one frequency coefficient such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and d) uses the encoded frequency coefficients to produce a compressed image data. The method of claim 44, wherein the frequency coefficients are selectively encoded, such that at least one of: a) defining each of the frequency coefficients to be encoded using a number of bits, and the value of the respective frequency coefficient Irrelevant; b) fewer bits are used to encode frequency coefficients corresponding to higher frequencies; c) progressively fewer bits are used to encode frequency coefficients corresponding to progressively higher frequencies; d) discard corresponding to At least one frequency coefficient of a higher frequency. The method of claim 44, wherein the method comprises applying a scaling factor to at least some of the frequency coefficients, thus encoding the scaled frequency coefficients, and wherein at least one of the following: a) applying different scaling factors to at least some of the frequency coefficients; b) applying the same scaling factor to each frequency coefficient; and c) the scaling factor is used to reduce the strength of each of the frequency coefficients. The method of any one of claims 44 to 46, wherein the method comprises: a) selecting one of a plurality of coding schemes; and b) encoding the pixel data using the selected coding scheme. The method of claim 47, wherein each of the coding schemes provides a degree of compression, and wherein the method comprises selecting the coding scheme based at least in part on at least one of: a) The degree of compression; and b) the location of the array of pixels. The method of any one of claims 44 to 48, wherein the method comprises selectively encoding the frequency coefficients according to at least one of: a) a selection rule; b) a desired degree of compression; and c) the one Or a location of the array of pixels within the plurality of images. The method of claim 49, wherein the method comprises: a) selecting one of a plurality of bit coding schemes; and b) encoding the frequency coefficients in accordance with the selected bit coding scheme. The method of claim 50, wherein each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different number of bits to provide different degrees of compression. The method of claim 50 or 51, wherein the bit coding scheme is selected based at least in part on at least one of: a) a selection rule; b) a desired degree of compression; and c) the location of the pixel array . The method of any one of claims 50 to 52, wherein the method comprises selecting the bit coding scheme according to at least one of: a) a transmission bandwidth of a communication link for transmitting the compressed image data; b) transmission service quality of a communication link for transmitting the compressed image data; c) movement of a display device; d) image display requirement; e) target display resolution; f) processed channel; a position of the pixel array within the one or more images; h) a position of the pixel array within the one or more images relative to an observer gaze point of the one or more images; And i) the error matrix. The method of any one of claims 50 to 53, wherein the frequency components are arranged in a plurality of levels, and wherein each of the bit coding scheme definitions is to be used to encode each of the plurality of levels The number of individual bits of the frequency coefficients. The method of claim 54, wherein the array is an N x N pixel array, resulting in a 2N-1 frequency component level. The method of any one of claims 44 to 55, wherein the method comprises: a) determining a gaze point of an observer of the one or more images; b) selectively selecting a frequency coefficient based at least in part on the gaze point coding. The method of claim 56, wherein the method comprises: a) determining a distance between the gaze point and a position of the pixel array in the one or more images; and b) selecting a frequency coefficient according to the distance Sexual coding, such a large distance has fewer frequency coefficients encoded. The method of any one of claims 44 to 57, wherein the image data defines a plurality of channels, and wherein the method comprises selectively encoding frequency coefficients for each channel. The method of claim 58, wherein the pixel data defines an RGB channel, and wherein the method comprises: a) converting the RGB channel to a YCbCr channel; and b) converting the YCbCr channels. The method of claim 59, wherein the method comprises at least one of: a) selectively encoding more frequency coefficients for the Y channel than the Cb or Cr channels; b) selectively targeting the YCbCr channels Simultaneously encoding more frequency coefficients; and c) generating the compressed image data by: i) encoding the CbCr channels; and ii) using the Y channel. The method of any one of claims 44 to 60, wherein the conversion is a two-dimensional (2-D) discrete cosine transform. The method of any one of claims 44 to 61, wherein the method comprises obtaining the pixel data from an image feed. The method of any one of claims 44 to 62, wherein the method comprises obtaining pixel data from the image data by: a) buffering image data corresponding to a next n -1 column of pixels of the image; b) buffering The next n pixels of image data of the next column of pixels; c) obtaining pixel data of the next n x n pixel block from the buffered image data; d) repeating steps b) and c) until all Pixel data is obtained in n columns of pixels; and e) steps a) through d) are repeated until pixel data has been obtained from each column of pixels of the image. The method of claim 63, wherein n : a) a selection rule is selected according to at least one of the following; b) a selected bit coding scheme; and c) the location of the pixel array. The method of any one of claims 44 to 64, wherein the method comprises: a) simultaneously selecting a coding frequency coefficient; and b) generating compressed image data at least in part by parallel to sequential byte encoding. An apparatus for compressing image data representing one or more images, the apparatus comprising at least one electronic encoder processing device, the at least one electronic encoder processing device: a) obtaining pixel data from the image data, the pixel material data representing a pixel array within the one or more images; b) applying a conversion to the pixel data to determine a set of frequency coefficients indicative of a frequency component of the pixel array; c) selectively encoding at least some of the frequency coefficients using a one-bit encoding scheme, thereby generating a set of encoded Frequency coefficient, wherein the bit coding scheme defines the number of bits used to encode the frequency coefficients, and wherein the frequency coefficients are selectively encoded: i) at least some of the encoded frequency coefficients have different bits Number ii) discarding at least one frequency coefficient such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and d) using the encoded frequency coefficients to produce compressed image data. The device of claim 66, wherein the device comprises: a) an encoder input buffer for receiving the image data; and b) an encoder output buffer for storing the compressed image data. The device of claim 66 or 67, wherein the device comprises an encoder input buffer, the encoder input buffer: a) buffering image data corresponding to the next n -1 column of pixels of the image; b Capturing image data of the next n pixels in the next column of pixels, allowing the at least one encoder processing device to acquire pixel data of the next n x n pixel block from the buffered image data; c) repeating step b), Until pixel data has been obtained from all of the n columns of pixels; and d) steps a) and b) are repeated until pixel data has been obtained from each column of pixels of the image. The device of any one of claims 66 to 68, wherein the device comprises an encoder transmitter that transmits the image data from the encoder output buffer. The apparatus of any one of claims 66 to 69, wherein the at least one encoder processing device comprises: a) a field programmable gate array edited by a suitable program; b) a dedicated integrated circuit; and c) a Graphics processing unit. The device of any one of claims 66 to 70, wherein the pixel data defines a plurality of channels, and wherein the device comprises: a) a separate processing device for each channel; and b) for simultaneous processing A parallel processing device for each channel. The device of claim 71, wherein the pixel data defines an RGB channel, and wherein the device: a) converts the RGB channel to a YCbCr channel; and b) selectively encodes the YCbCr channel using at least one processing device . The device of any one of claims 66 to 72, wherein the pixel data defines an RGB channel, and wherein the device: a) converts the RGB channels into CbCr channels using a YCbCr processing device; b) uses at least A respective processing device decodes the CbCr channels; and c) converting the Y channel from the YCbCr processing device to an output buffer using a delay block. The device of any one of claims 66 to 73, wherein the device comprises an encoding Communicating with a decoder by wireless communication, allowing image data to be transmitted as compressed image data between the encoder and the decoder. The device of claim 74, wherein the encoder is at least one of a processing system coupled to a suitable program editor and a portion thereof. The device of claim 74, wherein the decoder is coupled to at least one of a wearable display device and a portion thereof. The apparatus of any one of clauses 74 to 76, wherein the encoder communicates with the decoder to exchange at least one of: a) compressed image data; b) mobile data indicating a display device movement; c At least partially controlling the control data of the display device; d) indicating input data to the user input command; e) indicating gaze data of a viewer's gaze point; and f) from being associated with a wearable display device Sensor data for the sensor. The apparatus of any one of claims 66 to 77, wherein the apparatus performs the method of any one of claims 44 to 65. A method of decompressing compressed image data representing one or more images, the method comprising: a) acquiring compressed image data; b) defining one bit of the number of bits used in each encoded frequency coefficient a metacoding scheme, determining a set of encoded frequency coefficients from the compressed image data; c) performing bit decoding of the encoded frequency coefficients according to the bit encoding scheme, Generating a set of frequency coefficients, wherein at least one frequency coefficient is generated, such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and d) applying a reverse transform to the set of frequency coefficients to determine representative of the one or more images Pixel data for a pixel array. The method of claim 79, wherein the method comprises: a) selecting one of a plurality of decoding schemes; and b) decoding the pixel data using the selected decoding scheme. The method of claim 80, wherein the method comprises selecting the decoding scheme based at least in part on at least one of: a) a selection rule; b) a desired degree of compression; c) a location of the pixel array; d) an encoding code indicating one of the encoding schemes used, the encoding code being determined by the compressed data image. The method of claim 81, wherein the bit coding scheme uses a smaller number of bits to encode a frequency coefficient corresponding to a higher frequency, and wherein the method includes generating at least one corresponding to a higher frequency domain Some of these frequency coefficients. The method of any one of clauses 79 to 82, wherein the method comprises applying a scaling factor to at least some of the frequency coefficients, thereby transmitting the scaled frequency coefficients, and wherein at least one of the following: a Applying different scaling factors to at least some of the encoded frequency coefficients; b) applying the same scaling factor to each encoded frequency coefficient; c) The scaling factor is used to increase the strength of each of the encoded frequency coefficients. The method of any one of clauses 79 to 83, wherein the method comprises: a) selecting one of a plurality of bit coding schemes; and b) selecting the encoded frequency according to the selected bit coding scheme Coefficient decoding. The method of claim 84, wherein each of the plurality of bit coding schemes selectively encodes different frequency coefficients with different number of bits to provide different degrees of compression. The method of claim 84, wherein the bit coding scheme is selected based at least in part on at least one of: a) an encoding code; b) a selection rule; c) a desired degree of compression; and d) The location of the pixel array. The method of claim 86, wherein the selection rule depends on at least one of: a) a transmission bandwidth of a communication link for transmitting the compressed image data; b) for transmitting the compressed image data The quality of the transmission service of a communication link; c) the movement of a display device; d) the image display requirement; e) the resolution of a target display; f) the processed channel; g) within one or more of the images a position of the pixel array; h) the pixel array within the one or more images relative to the one or more images One observer gaze at one of the positions; and i) the error matrix. The method of any one of claims 79 to 87, wherein the method comprises: a) determining an observer's gaze point of the one or more images; b) at least partially encoding the encoded frequency coefficient according to the gaze point Selective decoding. The method of claim 88, wherein the method comprises: a) determining a distance between the gaze point and a position of the pixel array in the one or more images; and b) encoding the code according to the distance The frequency coefficient is selectively decoded, so that a larger distance produces more frequency coefficients. The method of any one of clauses 79 to 89, wherein the frequency components are arranged in a plurality of levels, and wherein each of the bit coding scheme definitions is to be used to encode each of the plurality of levels The number of individual bits of the frequency coefficients. The method of claim 90, wherein the array is an N x N pixel array, resulting in a 2N-1 frequency component level. The method of any one of clauses 79 to 91, wherein the image data defines a plurality of channels, and wherein the method comprises selectively decoding the encoded frequency coefficients for each channel. The method of claim 92, wherein the compressed pixel data defines a YCbCr channel, and wherein the method comprises: a) performing an inverse conversion of the YCbCr channels; and b) converting the converted YCbCr channels into RGB channel. The method of claim 92, wherein the method comprises at least one of: a) generating more frequency coefficients to the Cb or Cr channel than the Y channel; b) simultaneously decoding the encoded YCbCr channels; c) decoding the CbCr channels and converting the decoded CbCr channels to the Y channels into RGB channels. The method of any one of claims 79 to 94, wherein the inverse conversion is an inverse two-dimensional (2-D) discrete cosine transform. The method of any one of claims 79 to 95, wherein the method comprises using the pixel data to generate an image feed. The method of any one of clauses 79 to 96, wherein the method comprises: a) decoding the compressed image data at least in part by sequence to parallel byte decoding; and b) simultaneously selectively decoding the frequency coefficients. The method of any one of claims 44 to 65 or 79 to 97, wherein the digital reality is at least one of: a) augmented reality; b) virtual reality; and c) mixed reality . The method of claim 44, wherein the method is for receiving the compressed image data from a computing device by using at least one of the following: Displaying video data in a digital reality headset: a) a communication network; b) A wireless communication link. The method of any one of claims 44 to 65 or 79 to 99, wherein the method is used for at least one of: a) transmitting virtual reality video material; and b) wirelessly transmitting virtual reality video material . An apparatus for decompressing compressed image data representing one or more images, the apparatus comprising at least one electronic decoder processing device: a) acquiring compressed image data; b) Defining a one-bit encoding scheme for the number of bits used in each encoded frequency coefficient, determining a set of encoded frequency coefficients from the compressed image data; c) performing the encoded frequency coefficients in accordance with the bit encoding scheme Bit decoding, thereby generating a set of frequency coefficients, wherein at least one frequency coefficient is generated such that the set of encoded frequency coefficients is less than the set of frequency coefficients; and d) applying a reverse transform to the set of frequency coefficients to determine Pixel data of a pixel array within one or more images. The device of claim 101, wherein the device comprises: a) a decoder input buffer for receiving the image data; and b) a decoder output buffer, wherein the image data is stored. The device of claim 102, wherein the device comprises a decoder transceiver that receives the compressed image data and provides the compressed image data to the input buffer. The apparatus of any one of claims 101 to 103, wherein the at least one decoder processing device comprises at least one of the following: a) a field programmable gate array; b) a dedicated integrated circuit; and c) a graphics processing unit. The apparatus of any one of claims 101 to 104, wherein the compressed image material defines a plurality of channels, and wherein the device comprises at least one of: a) a separate processing device for each of the channels And b) a parallel processing device for processing each channel simultaneously. The device of claim 105, wherein the compressed image material defines a TCbCr channel, and wherein the device: a) decodes the CbCr channels using at least one processing device; and b) converts the decoded YCbCr channels Into the RGB channel. The apparatus of claim 105, wherein the compressed image material defines a YCbCr channel, and wherein the device: a) decodes the CbCr channels using at least one processing device; b) uses an RGB processing device, The decoded CbCr channel and the Y channel are converted to RGB channels; and c) the Y channel is converted from a decoder input swap region to the RGB processing device using a delay block. The device of any one of claims 101 to 107, wherein the device comprises a decoder for communicating with an encoder by means of wireless communication, allowing image data to be used as compressed image data in the encoder and the decoder Transfer between. For example, the device of claim 108, wherein the decoder is coupled to a suitable program At least one of the computer systems and part of it. The device of claim 108, wherein the decoder is coupled to at least one of a wearable display device and a portion thereof. The device of any one of claims 101 to 110, wherein the decoder communicates with the encoder to exchange at least one of: a) compressed image data; b) mobile data indicating a display device movement; c At least partially controlling the control data of the display device; d) indicating input data to the user input command; e) indicating gaze data of a viewer's gaze point; and f) from being associated with a wearable display device Sensor data for the sensor. The apparatus of any one of claims 101 to 111, wherein the apparatus performs the method of any one of claims 79 to 100. A method of compressing image data representing one or more images, the method comprising: a) obtaining pixel data from the image data, the pixel material data representing a pixel array within the one or more images; b) determining An encoding scheme; c) encoding the pixel data using the encoding scheme; d) determining an encoding code indicating the encoding scheme used; and e) using the encoding code and the encoded pixel data to generate compressed image data. The method of claim 113, wherein the method comprises determining the coding scheme by at least one of: a) based on an image type of the image data; b) based on a coding scheme indication received from an image data source; c) utilizing at least one of the image data and the pixel data; and d) including at least A compression requirement: i) a total amount of compression; ii) a resulting image quality; and iii) a compression delay. The method of claim 114, wherein the method comprises analyzing the pixel data to determine whether the pixel array is at least one of: a) a gradient; b) a boundary; and c) a single color. The method of any one of claims 113 to 115, wherein the method comprises at least one of the following: a) if the pixel array is a solid color, replacing the pixel array with an encoding code representing the solid color; b) If the pixel array is a gradient, the method comprises encoding the pixel data using a method as in claim 60 of the patent application; and c) using any one of claims 1 to 21 or 44 to 65 The method of the item encodes the pixel data. An apparatus for compressing image data representing one or more images, the apparatus comprising at least one electronic encoder processing device, the at least one electronic encoder processing device: a) obtaining pixel data from the image data, the pixel material data representing a pixel array within the one or more images; b) determining a coding scheme; c) encoding the pixel data using the coding scheme; d) determining An encoding code indicating the encoding scheme used; and e) generating compressed image data using the encoded frequency coefficients. The apparatus of claim 117, wherein the apparatus performs the method of any one of claims 113 to 116. A method of decompressing compressed image data representing one or more images, the method comprising: a) acquiring compressed image data; b) determining an encoded code from the compressed image data; c) using the encoded code Determining a coding scheme; and d) decoding the compressed image data using the coding scheme to determine pixel data representing a pixel array within the one or more images. The method of claim 119, wherein the method comprises at least one of: a) replacing an encoded code with a solid color pixel array; b) using the method of claim 94, the compressed image The data is decoded into a gradient of a pixel array; and c) the compressed image data is decoded using a method as claimed in any one of claims 21 to 39 or 79 to 96. A device for decompressing compressed image data representing one or more images, the device package Comprising at least one electronic decoder processing device, the at least one electronic decoder processing device: a) acquiring compressed image data; b) determining an encoding code from the compressed image data; c) determining an encoding scheme using the encoding code And d) decoding the compressed image data using the encoding scheme to determine pixel data representing a pixel array within the one or more images. The apparatus of claim 121, wherein the apparatus performs the method of any one of claims 119 and 120.