KR102467139B1 - Data processing systems - Google Patents

Abstract

The data processing system 1 comprises a display controller 7 operative to provide the display 12 with a compressed version of the output surface to be displayed that has been compressed using a lossy compression scheme, and a display controller 7 used in forming the output surface to be displayed. and one or more processing stages 2-6 operative to provide a surface or surfaces. At least one of the one or more processing stages 2-6 is a surface or one or more regions of surfaces, provided by the processing stage, used in forming an output surface to be displayed in compressed form using a lossy compression scheme. works to output them.

Description

Data processing system {DATA PROCESSING SYSTEMS}

The present invention relates to a data processing system, and more particularly to the processing of data when generating a surface for display on a display in a data processing system.

In a data processing system, an image to be displayed will generally be processed by a number of processing stages before it is finally displayed on a display.

For example, one or more generators such as graphics processing units (GPUs), video engines, etc. will generate frames (faces) that contribute to the final displayed face.

Also, the composition engine may combine a plurality of individual input surfaces into a single synthesized output surface.

The final face to be displayed (e.g. the composited frame) is usually stored in a frame buffer in memory, and from that frame buffer the display controller for the display (e.g. by internal DMA (Direct Memory Access)). is read The display controller then passes the frame (eg, through the pixel pipeline) to a display for display (the display may be a screen or a printer, for example).

The bandwidth cost of sending pixel data from the display controller to the display can be significant. One way to reduce the bandwidth and power required in the data processing system is to compress the pixel data sent from the display controller to the display. For example, the display controller uses Display Stream Compression (DSC) (designed to be a compression scheme that is mathematically lossy, but "visually lossless" (i.e., invisible to the user)). It is also possible to compress the pixel data passed to the display from

Although this compression technique has been successful in reducing the power and bandwidth requirements of the system, Applicant believes that room for improvement remains for such data processing systems.

According to a first aspect of the present invention, as a data processing system,

a display controller operative to provide a compressed version of an output surface to be displayed to a display that has been compressed using a lossy compression scheme;

A data processing system is provided, having one or more processing stages operative to provide a surface or surfaces to be used in forming an output surface to be displayed;

At least one of the one or more processing stages is configured to output a surface or one or more regions of surfaces, provided by the processing stage, used in forming an output surface to be displayed in compressed form using a lossy compression scheme. works,
the display controller includes a compression stage and a decompression stage;
The data processing system is configured such that one or more regions output by at least one of the one or more processing stages in a compressed form using a lossy compression method bypass the display controller compression stage and decompression stage. do.

According to a second aspect of the invention, a display controller operative to read a face or faces from a memory and provide a compressed version of an output face to be displayed to a display that has been compressed using a lossy compression scheme, and an output face to be displayed are provided. CLAIMS 1. A method of operation of a data processing system having one or more processing stages operative to provide a surface or surfaces to be used in forming, the method comprising:

one or more of said one or more processing stages providing a surface or surfaces used in forming an output surface to be displayed;

reading at least one of the faces presented by the processing stage from memory, using this face to form an output face to be displayed, and providing the output face to a display for display in compressed form using a lossy compression scheme; having the display controller;

at least one of the one or more processing stages outputs a surface or one or more regions of surfaces, provided by the processing stage, used in forming an output surface to be displayed in compressed form using a lossy compression scheme;
the display controller includes a compression stage and a decompression stage;
The method of operation includes one or more regions output by at least one of the one or more processing stages in compressed form using a lossy compression scheme that bypasses the display controller compression stage and decompression stage.

The present invention relates to a data processing system in which a display controller is operative to provide an output face to a display that has been compressed using a lossy compression scheme. This data processing system also includes one or more processing stages upstream of the display controller that provide the surfaces used in forming the displayed surface, but in contrast to known arrangements, at least one of the upstream processing stages is lossy compressed. It is operated to output the surface provided by the upstream processing stage in compressed form, using .

Thus, in the present invention, at least one surface used to generate the output surface to be displayed is compressed using a lossy compression scheme upstream of the display controller. This means that the overall memory bandwidth and power requirements of the system can be reduced.

Additionally, the applicant has recognized that this can be done without adversely affecting the quality of the displayed image. This is because the lossy compression used to compress the output surface presented to the display from the display controller effectively discards some of the image quality, so in many cases, some of the image quality is instead efficiently stored at an early (upstream) stage in processing. It is not a problem if it is discarded.

Accordingly, the present invention relates, at least in part, to an arrangement in which an output surface provided to a display from a display controller is compressed using a lossy compression method (such as DSC), wherein an output surface for display is displayed at an initial (upstream) stage in processing. It is to realize saving power and bandwidth without adversely affecting the quality of a displayed image by lossy compressing the data used to generate .

In the data processing system of the present invention, one or more processing stages provide surfaces used to form an output surface to be displayed, and a display controller provides the output surface (in lossy compressed form) to a display for display. .

The face or faces provided by the processing stages may be any suitable desired face(s) (eg frames). Preferably, they are windows to be displayed, preferably pictures for display, eg frames. These faces may be, for example, for games, demos, graphical user interfaces (GUIs), GUIs with video data (eg, video frames with graphical "play" and "pause" icons).

The output surface to be displayed may correspondingly have any suitable desired such surface (eg frame) for display. Again, this is preferably a window, preferably a picture, eg a frame, to be displayed.

The one or more processing stages operative to provide the surface or surfaces used to form the output surface to be displayed may include any suitable processing stage(s) in a data processing system operable to provide display surfaces. may be They will be, and preferably are, "upstream" of the display controller in the overall processing flow of generating and presenting the displayed output surface. There may be only one processing stage upstream of the display controller, but preferably there are multiple processing stages.

The processing stages may include at least one processing stage operative to generate a display surface. Thus, in one preferred embodiment, at least one of the one or more processing stages includes a frame generator operative to generate one or more planes. The “frame generator” processing stage(s) may include, for example, a graphics processing unit (GPU), a video codec or video engine, a digital camera image signal processor (ISP), an image processor, and/or a central processing unit (CPU), etc. You may. If desired, there may be more than one frame generator.

The frame generator may suitably encode the input in the case of a video codec or video engine from a captured picture in the case of a digital camera image signal processor (ISP) or the like, in a suitable manner, for example by rendering a facet in the case of a graphics processor. decoded video data, it must generate its face.

The one or more processing stages may also or instead (preferably also) include a stage or stages operable to process a previously generated face or faces, for example to create a modified version of the face or facets. have.

This would be the case, for example, when the composition engine composites multiple faces to provide a composite face.

Thus, in a preferred embodiment, at least one of the one or more processing stages includes a composition stage operable to synthesize (two or more) faces to generate a composite face. In this case, it is preferable that the faces synthesized by the composition stage have (two or more) faces generated by at least one frame generator. Accordingly, the composition stage is preferably operated to read (two or more) faces from memory.

The composition stage may be configured to composite faces as desired in any suitable manner to generate a single composite face. In one embodiment, the composition stage is configured to blend or otherwise combine faces to generate a single composite face. The composition stage is preferably operable to store the synthesized output surface in memory. In a preferred embodiment, the composited face should be used as the displayed output face.

The one or more processing stages may also or alternatively (preferably also) include a compression stage operative to compress the input surface to provide a compressed display or version of the input surface. Thus, in one embodiment, at least one of the one or more processing stages comprises a compression stage operable to compress (one or more areas of) a surface. The compression stage(s) may be operable to compress one or more or all regions of one or more or all faces produced by other processing stages of the system.

Again, the compression stage is preferably operative to store the compressed face or faces in memory.

One or more processing stages may also or instead (preferably also) be operated to (optionally) deform one or more regions or surfaces. For example, one or more stages of processing may change image enhancement to one or more regions or planes, such as changing contrast and/or brightness, performing gamma correction, and/or performing tone mapping. (optionally) perform image enhancement modifications; (optionally) rotate and/or scale one or more regions or faces; and/or (optionally) perform a picture enhancement scheme, such as, for example, Level Adaptive Overdrive (LAO), Feed Forward Drive (FFD), Dynamic Contrast Compensation (DCC), etc., to improve display responsiveness. have. One or more processing stages may, of course, be operated to perform other types of processing and/or changes.

A preferred embodiment comprises one or more frames, for example generating processing stages which generate the frames that are preferably used to form the output surface to be displayed (processing that generates the output surface to be displayed). There is a series of processing stages (in order of flow), followed by taking the faces created by the frame generator and processing one or more of these faces (e.g., compositing them to form a composite face), e.g. For example, there is one or more processing stages, such as a composition stage, which form the output surface to be displayed and/or form the additional surface or surfaces that should contribute to the output surface. The display controller may then take one or more or all of the surfaces created by the processing stages to form an output surface that the processing stages pass to the display.

Accordingly, a surface created by a processing stage used in forming an output surface to be displayed may be used directly as, or as part of, an output surface to be displayed, and/or a process used in forming an output surface to be displayed. The face or faces produced by a stage may not be used as or directly on an output face to be displayed, but instead may be initially processed by another processing stage, the output of which is displayed. It is used as an output surface or as part of an output surface (or is further processed, etc.). In the latter case, the facets created by the processing stage that must be used when forming the output facet are in fact generating the facet or facets used to form and/or contribute to the final output facet displayed. It will be the middle plane in the whole processing flow.

As described above, the processing stages are preferably operated to store their respective output faces to memory (and correspondingly to read the faces they are to process from memory (where appropriate)). This memory may comprise any suitable memory and may be configured in any suitable and desired manner. For example, it may be a memory that is on chip with and/or local to the processing stage in question, or it may be an external memory. In a preferred embodiment, it is an external memory, such as the main memory of the data processing system. It may also be dedicated memory for this purpose, or it may be part of memory used for other data as well. In a preferred embodiment, one or more faces are stored in (read from) one or each frame buffer. For example, each processing stage frame buffer may be located in the main memory of the data processing system.

The display controller may comprise any suitable display controller operable to present to the display an output surface that has been compressed using a lossy compression scheme.

As such, the display controller preferably reads the face or one or more regions of faces to be displayed, from a memory in which the face or faces are stored, and provides a lossy compressed output face formed from the input face or faces to the display for display. It will operate to do, and can be operated preferably.

The display controller could be operated to simply read a single input plane providing all or some of the lossy compressed output planes to the display, or alternatively, the display controller could (if desired) read and combine multiple input planes. By doing so, it can be operated to provide a lossy compressed output surface to a display. As explained above, the (input) faces that the display controller reads and uses to provide the output face to the display will be provided by one or more of the (upstream) processing stages of the data processing system. Thus, the input surface to the display controller is one or more areas of one or more surfaces generated by the frame generator or generators, and/or one or more areas of a composite output surface consisting of a composition stage or stages. fields, and/or one or more regions of one side output by the compression stage or stages, and the like.

Accordingly, the display controller preferably includes an input stage operable to read the input surface or one or more regions of the surfaces. In a preferred embodiment, the input stage includes a read controller, for example a DMA (Direct Memory Access) read controller.

A display controller is operative to provide a lossy compressed output surface to a display. As such, the display controller preferably includes an output stage operable to provide a lossy compressed output surface to the display. This output stage may be any suitable such output surface operative to provide a display output surface to a display, for example to cause the display output surface to be displayed on the display (to act as a display interface). The output stage preferably includes appropriate timing control functions for the display (eg it is configured to send pixel data to the display with appropriate horizontal and vertical blanking periods).

To enable providing a lossy compressed output surface to a display, the display controller can and is preferably operable to compress one or more regions of one or more surfaces using a lossy compression scheme. The display controller preferably includes a compression stage operable to compress one or more areas of one or more faces for this purpose.

The lossy compression scheme that the display controller may use to compress the input surface or surfaces to provide a compressed output surface may include any suitable desired lossy compression scheme.

The display controller preferably uses a substantially visually lossless compression scheme, preferably Delta Pulse Code Modulation (DPCM) and/or Indexed Color History (ICH). The display controller lossy compression scheme preferably operates on the raster lines of the plane to be compressed, ie each raster line (row) of pixels (sampling positions) is compressed together. Most preferably, the display controller uses and supports Display Stream Compression (DSC).

The display controller shall be operable to compress any uncompressed input surfaces or input surface regions to be presented to the display as lossy compressed output surfaces using an appropriate lossy compression scheme (prior to presenting the surface or surface area to the display) and , preferably works.

As described above, as well as potentially receiving uncompressed (input) surfaces (and/or (input) surface regions) for presentation to a display, the surfaces that should form or contribute to an output surface ( and/or regions of these faces) may be provided in compressed form (whether in lossy compressed form or otherwise). That is, the display controller input surface or surfaces may be compressed or uncompressed and/or may have one or more regions that are compressed and/or one or more regions that are not compressed.

If the (input) planes or plane regions are already compressed, the display controller may or may not need to convert these compressed planes (or plane regions) to a lossy compressed form that must be passed on to the display. may be

For example, if the input surface is already in the lossy compressed form needed for the display, the display controller can pass the already compressed input surface (or input surface region or regions) to the display without further compression, and preferably simply passes.

On the other hand, if the compression scheme used for the input plane or planes does not match the lossy compression scheme used for the output plane presented to the display, the display controller will properly provide the compressed input plane to the display. It needs to, and preferably does, convert to a lossy compressed format. This may entail decompressing the compressed input face or faces (region or regions), then recompressing the decompressed face or faces (region or regions) using a display controller lossy compression scheme, preferably It entails

As such, in a preferred embodiment, the display controller decompresses the compressed regions of the face to be compressed using a compression method other than the display controller lossy compression method (eg, other than DSC), such that, for example, this face It may be and is operative to generate one or more decoded and/or decompressed regions of The display controller preferably includes a decompression stage operable to decode and/or decompress one or more regions of a face for this purpose. The display controller decompression stage may include, for example, a DSC decoder for decompressing one or more DSC compressed regions of a side and/or an ARM frame buffer for decompressing one or more AFBC compressed regions of a side. It may also have a compression (AFBC) decoder (AFBC is described in US-A1-2013/0034309).

This display controller (compression stage) is preferably operative to compress the decompressed region(s) (using a display controller lossy compression scheme) to generate one or more regions, eg of an output surface.

Thus, the display controller (its compression stage) can (e.g., by means of the display controller decompression stage) uncompressed regions of one (input) side and/or regions of one (input) side that have not been compressed (e.g. by the display controller decompression stage). optionally) is preferably operated to compress.

Thus, in one embodiment, the lossy compressed output surface presented to the display is one or more regions that have been compressed by the display controller (eg, its compression stage), and/or may have one or more regions that have been compressed by other than (eg, by one or more of the one or more upstream processing stages, etc.)

Thus, in one embodiment, the output surface (which has been compressed using the "display controller" lossy compression scheme) presented to the display is compressed (preferably by one or more than one) other than by (and not by) the display controller. one or more regions of the surface) compressed by one or more of the processing stages.

In one embodiment, the display controller performs some form of processing on the input surface or surfaces used to form the displayed output surface, for example (optionally) converting one or more regions of the input surface or surfaces. works to change The display controller may have one or more processing stages operable to process an input surface to generate an output surface to be displayed for this or that purpose.

The display controller may (optionally) make picture enhancement changes to the input surface or one or more regions of surfaces, such as, for example, changing contrast and/or brightness, performing gamma correction, and/or performing tone mapping. ) may be operated to do. Additionally or alternatively, the display controller may be operated to (optionally) rotate and/or scale the area or face. In addition or in dash, the display controller (optionally) performs a picture enhancement scheme to respond to the display, e.g., Level Adaptive Overdrive (LAO), Feed Forward Drive (FFD), Dynamic Contrast Compensation (DCC), etc. It can also work to improve sexuality. The display controller may of course also be operable to perform other types of processing and/or changes.

In these embodiments, the display controller selectively decodes and (i.e., if necessary) one or more regions of the input surface that are encoded and/or compressed so that the regions can be processed (changed by the display controller). /or may decompress, and in a preferred embodiment decode and/or decompress. The display controller (its compression stage) is preferably operative to (lossy) compress the altered region(s), eg to generate one or more regions of the output surface that are lossy compressed to be displayed.

In a preferred such embodiment, when an already compressed (input) surface area is modified by the display controller, whether the modified version is equal to or sufficiently similar to the unmodified version of the input surface area, and thus the unaltered compression of the area Determine if the modified version is output by the display controller (ie instead of the modified, compressed version). In this embodiment, when the modified version of the region is output by the display controller, the modified version of the region does not need to be compressed (preferably not compressed) by the display controller, so that the display controller does not need to. throughput can be reduced.

A region that has been changed is defined by its It may also be determined to be sufficiently similar to the corresponding unaltered area.

Thus, in these embodiments, when a region is changed, either a changed version of the region or an unaltered version of the region is output by the display controller, depending on whether the change causes a sufficient change to the region. (i.e. presented to the display). When an unchanged version of the region is output by the display controller, the unchanged version of the region is preferably discarded and does not need to be compressed (preferably uncompressed) by the display controller, so that the display controller The amount of processing required can be reduced.

In a particularly preferred embodiment, the display controller is configured such that these region(s) on the (input) side are already compressed using the "display controller" lossy compression scheme and no changes are required, if any. It is configured to be able to bypass the decompression stage, the alteration stage and the compression stage. The region(s) of such (input) face is then preferably used as an output face or as part of an output face.

Thus, in a preferred embodiment, the display controller includes preferably all of one or more of a compression stage, one or more modification stages, and one or more decompression stages, each of these stages (if present) preferably yes. For example, it can be selectively bypassed when appropriate or desired. This means that the amount of processing done by the display controller can be reduced.

Thus, in a preferred embodiment, the display controller reads one or more regions of the input surface; It is configured to determine whether one or more of the one or more read regions are compressed.

In one embodiment, if it is determined that the read area is not compressed (i.e., not compressed), the display controller may, in order to generate an appropriate lossy compressed output surface area for presentation to the display, display controller: It is preferred to compress the read area (which, if desired, may be modified before being compressed) using a lossy compression scheme, for example DSC.

On the other hand, in one embodiment, if it is determined that the read area is a compressed area, the display controller determines whether the read area is compressed using a display controller lossy compression scheme (e.g., DSC). desirable. If (if) it is determined that the read area is compressed using a compression method other than the display controller lossy compression method, the display controller decompresses the read area and compresses it using a display controller lossy compression method (e.g., DSC). It is desirable to compress the released area.

This compressed area is preferably used for the (corresponding to) area of the output surface presented to the display.

Again, the decompressed region may be modified before being compressed, if desired.

Correspondingly, in one embodiment, if (if) it is determined that the read area is compressed using a display controller lossy compression method (e.g., DSC), the display controller determines that the read area is compressed using a display controller lossy compression method. It is desirable to determine whether a region requires a change.

In one embodiment, if it is determined that the read area that is compressed using the display controller lossy compression method does not require modification (i.e., does not require modification), the display controller, as the output surface, Alternatively, as part of the output surface, it is preferable to provide the display with a readout area that is compressed using a display controller lossy compression method.

On the other hand, if it is determined that the read area compressed using the display controller lossy compression method requires modification, the display controller decompresses the read area compressed using the display controller lossy compression method, and compresses the read area compressed using the display controller lossy compression method. It is preferable to change the released area and determine whether it is the same as or sufficiently similar to the read area compressed using the display controller lossy compression method.

If it is determined that the altered region is equal to or sufficiently similar to the read region to be compressed using the display controller lossy compression scheme, the display controller preferably uses the display controller lossy compression scheme as the output plane or as part of the output plane. It is preferred to provide the display with a read region that is compressed using , and the altered region is preferably discarded.

On the other hand, if it is determined that the changed area is not the same as or not sufficiently similar to the read area compressed using the display controller lossy compression method (i.e., not equal to or not sufficiently similar to the read area (i.e., sufficiently different)) ( surface), the display controller preferably compresses the altered area using a display controller lossy compression scheme, and then the so compressed area is then used for the corresponding area of the output surface, i.e., the so compressed The area is preferably provided on the display, preferably as an output surface or as part of an output surface.

In one embodiment, the display controller is operative to write the output face to external memory. The display controller may have a write-out stage operable to write the output surface to an external memory for this purpose. The light out stage may be any suitable such stage capable of writing the output surface to external memory. In one embodiment, the light out stage includes a write controller, such as a Direct Memory Access (DMA) write controller.

In a preferred embodiment, the output surface that may be written to external memory is (optionally modified version of) an output surface provided on a display.

The external memory to which the light out stage can write data must be at least one memory external to the display controller, and preferably at least one memory. The external memory is preferably installed as or on a separate chip (monolithic integrated circuit) in the display controller. The external memory preferably includes a main memory (eg, shared with a central processing unit (CPU)) or a frame buffer of the main memory, eg, the entire data processing system.

At least one of the one or more (upstream) processing stages of the data processing system may output one or more regions of a surface that the processing stage generates (using a lossy compression scheme) in lossy compressed form. Any one or more of the one or more upstream processing stages may be configured to do this.

Accordingly, in a preferred embodiment, the system comprises one or more frame generator processing stages operative to output one or more regions of one or more faces that have been compressed using a lossy compression scheme; a composition processing stage operative to output one or more regions of the synthesized output surface that have been compressed using a lossy compression scheme; and one or more compression processing stages operative to output one or more regions of one or more faces that have been compressed using a lossy compression scheme.

In one embodiment, only one of the one or more processing stages is operated to output a face that has been compressed using a lossy compression scheme. However, there may be multiple stages of one or more processing stages capable of outputting (outputable) one or more regions of a surface that have been compressed using a lossy compression scheme (in a preferred embodiment, there are multiple stages). ). Each processing stage capable of providing a compressed output preferably includes (e.g., executes a compression routine) an appropriate compression "engine" for that purpose.

The processing stage(s) capable of providing compressed output may be selected as desired.

In general, it is desirable that the compressed output should be provided as early as possible in the overall processing flow of generating and presenting the displayed output surface, since this means that bandwidth and power are reduced as much as possible.

As such, preferably at least one of the frame generator processing stage(s) is capable of outputting one or more regions of one or more surfaces that have been compressed using a lossy compression scheme.

In embodiments using composition, preferably at least the composition stage can output one or more regions of the composited output surface that have been compressed using a lossy compression scheme. The composition stage preferably has a compression engine operable to compress the synthesized surface using a lossy compression method for this purpose.

When one or more of the compressed regions or surfaces output by a processing stage is further modified and/or processed (eg by one or more other processing stages), i.e. to be presented to a display controller. Prior to and/or being used as or to form an output surface for display, one or more compressed regions or surfaces (e.g., in one or more processing stages that undergo modification and/or processing) It must be decompressed (in the embodiment, decompressed) before being appropriately modified and/or processed (preferably recompressed).

For example, in an embodiment using composition, and if one or more of the regions or planes to be synthesized is compressed using a lossy compression method, one or more of the compressed regions or planes to be synthesized and Preferably all are decompressed prior to compositing, eg if desired or necessary for proper operation of the composition stage.

As such, in a preferred embodiment, one or more of the processing stages (preferably at least a composition stage) decompresses compressed regions of the surface, for example (e.g., for modification and/or compositing). ) can be, and is operated to generate, one or more decompressed regions of the surface. One or more of the one or more processing stages (eg the composition stage) preferably includes a decompression engine operable to decompress one or more regions of a face for this purpose.

In embodiments where one or more compressed region(s) are decompressed, the bandwidth requirements of the system are still advantageously reduced, i.e. for the particular case where data is transmitted between processing stages, at least in compressed form. point should be kept in mind.

In some embodiments, the area(s) or facets that need to be altered and/or processed and/or synthesized, for example where the benefit of bandwidth reduction is outweighed by the need to do additional compression/decompression processing. s), it may be desirable to avoid the processing required to compress, decompress (and then recompress). In such cases, if it is known, for example, that the region(s) or face(s) are to be altered and/or treated and/or synthesized, further processing may be performed to transform the region(s) or face(s), for example. It can be avoided by not compressing it initially.

Thus, in one embodiment, the area or surface provided by the processing stage is further modified and When processing and/or compositing, the processing stage is preferably controlled to output the region or plane in uncompressed form. Correspondingly, the area or surface provided by the processing stage is further modified and/or processed and/or before being presented to the display controller and/or before being used as or to form an output surface for display. Or, if not synthesized (not synthesized), the processing stage is preferably controlled to output one or more regions or faces in compressed form using a lossy compression method.

One or more of the processing stage(s) capable of providing compressed output may be configured to always provide compressed output (but correspondingly one or more of the processing stage(s) providing uncompressed output). may be configured to always provide this uncompressed output), but more preferably one or more of the processing stage(s) provides both compressed and uncompressed output as desired, preferably selectively. can do. Such a processing stage is preferably operated to selectively enable/disable its compression operation to do this.

Most preferably, a processing stage operable to provide both compressed and uncompressed output is operable to selectively provide compressed or uncompressed output on a region-by-region or face-by-area basis.

Thus, in one embodiment, for each area or surface output from a (specific) processing stage, before the area or surface is presented to the display controller and/or as or forming the output surface for display. It is determined whether it is modified and/or processed and/or synthesized before being used for

Preferably, when a region or facet is determined to be altered and/or processed and/or synthesized (face), the processing stage is preferably operated to output the region or facet in uncompressed form. Preferably, if it is determined that the region or facet is not altered and/or processed and/or synthesized (face), the processing stage is preferably operated to output the region or facet in compressed form.

This can help ensure that overall power consumption of the system due to bandwidth and processing requirements is reduced.

In another embodiment, any necessary change(s) to an area or surface can be predicted (e.g., normally by a display controller) before passing the area or surface through an entire sequence of stages, for example. to improve display responsiveness (e.g., Level Adaptive Overdrive (LAO), Feed Forward Drive (FFD), Dynamic Contrast Compensation (DCC), etc.) (e.g., relatively simple) image processing changes, such as performing gamma correction, and/or tone mapping; rotation and/or scaling; and/or image enhancement schemes. , the processing stage presenting the area or surface may be operable to perform its own image processing modifications.

Any one or more of the one or more upstream processing stages may be configured to (optionally) perform image processing changes to one or more regions or surfaces in this manner. As such, for example, preferably at least one of the frame generator processing stage(s) may perform image processing changes. In embodiments using composition, preferably at least the composition stage can perform the image processing changes.

One or more of the processing stage(s) always capable of performing such image processing changes may be configured to always perform such image processing changes, but more preferably one or more of the processing stage(s) are as desired. , can selectively perform image processing changes. Most preferably, this can be done area by area or area by area.

In one embodiment, for each region or facet output from a (specific) processing stage, the region or facet is, ie, before being presented to a display controller and/or as or forming an output facet for display. before being used for the purpose of determining whether it is (or is likely to be) altered and/or processed and/or synthesized.

Preferably, if it is determined that the region or surface is not (not) subject to further alteration and/or processing and/or compositing, the processing stage is operative to perform its own desired image processing alteration(s). it is desirable to be Preferably, if an area or surface is determined to be further altered and/or processed and/or composited (a surface), the processing stage is preferably operated so as not to perform its own desired image processing change(s). .

This means that the altered (and compressed) regions or faces do not need to be decompressed by the display controller to perform image processing changes, so that the display controller decompression, modification, and/or compression stage(s) can be bypassed. may mean that the processing required to compress, decompress (and recompress) the region(s) or surface(s) may be avoided.

Each (upstream) processing stage capable of providing a lossy compressed output surface may use the same or a different lossy compression scheme than the other processing stages capable of providing a lossy compressed output surface. However, in a preferred embodiment, each (upstream) processing stage capable of providing a lossy compressed output surface uses the same lossy compression scheme as the other processing stages capable of providing a lossy compressed output surface.

The lossy compression scheme or schemes used by the upstream processing stages may include any suitable desired lossy compression scheme.

In a preferred embodiment, the lossy compression scheme used by the (upstream) processing stages, or each lossy compression scheme, is based on the lossy compression scheme used by the display controller (used to compress the output face to the display). selected) method. The “processing stage” compression scheme is in some way (eg, preferably with respect to one or more of its characteristics) related to, and/or matches, a display controller lossy compression scheme. it is desirable to be

Thus, for example, in a preferred embodiment, the processing stage lossy compression scheme uses the same compression format and/or algorithm as the display controller lossy compression scheme, and the display controller compression scheme is a linear compression scheme (by comparison, e.g. For example, if it is a block-based compression method), for example, it is a linear compression method. Preferably, the processing stage lossy compression scheme is a linear compression scheme.

As will be appreciated by those skilled in the art, lossy compression schemes effectively discard some of the image quality in order to compress a picture. Lossy compression schemes are usually able to compress data to a greater degree than lossless compression schemes, but at the expense of introducing noise or artifacts into the image data. Thus, a given lossy compression scheme will have its own visual effect on the picture, eg its own specific noise characteristics or artifacts.

By matching the lossy compression schemes used, the overall memory bandwidth and power requirements of the system can be reduced without adversely affecting the quality of the displayed picture.

In a particularly preferred embodiment of the present invention, the noise characteristics of the processing stage lossy compression scheme(s) are selected (matched) based on the noise characteristics of the display controller compression scheme, preferably using two lossy compression schemes. So the compressed faces will show the same or similar noise artifacts.

In one embodiment, the lossy compression schemes are matched by the processing stage(s) using the same lossy compression scheme as the display controller.

As such, the lossy compression scheme used by the processing stage(s) may include a substantially visually lossless compression scheme, and may utilize Delta Pulse Code Modulation (DPCM) and/or Indexed Color History (ICH). A lossy compression scheme may operate on a raster line of an image to be compressed, i.e. where it is desired that each raster line (row) of pixels (sampling positions) be compressed together. In a preferred embodiment, the processing stages use Display Stream Compression (DSC).

In another embodiment, the processing stage(s) use a different lossy compression scheme for the display controller, but the lossy compression schemes (preferably to try to match the noise characteristics of the display controller compression scheme) display controller compression Matched by the processing stage compression method(s) based on the method, preferably configured based on the characteristics of the display controller compression method.

In these embodiments, it is preferred that the processing stage lossy compression scheme be based as closely as possible to the display controller compression scheme (e.g., DSC), ie to ensure that the noise characteristics are as similar as possible. Thus, for example, processing stage lossy compression schemes preferably use the same type of compression (eg, Delta Pulse Code Modulation (DPCM) and/or Indexed Color History (ICH)) if possible.

In one embodiment, the processing stage(s) providing compressed output is configured to provide compressed output regions or surfaces in a form (directly) compatible with display controller lossy compression schemes. For example, if the display controller lossy compression scheme is configured to operate on raster lines on the output side (e.g., a frame for display), then the processing stage(s) that provide the compressed output will be used on the raster lines, i.e. sampling. It can be configured to provide compressed output regions in the form of rows of locations (eg pixels).

As such, in one embodiment, the processing stage(s) that provide the compressed output have a size (e.g., preferably a sampling location) corresponding to the size and shape of the regions the display controller compression scheme is configured to operate on. (i.e. at least one of the one or more processing stages) The compressed regions output by one have a size and shape corresponding to the size and shape of the compressed regions of the output surface that the display controller presents to the display).

That is, in this embodiment, the compression "window" within which the processing stage lossy compression scheme operates will be the same compression window within which the display controller lossy compression scheme operates.

This makes the compressed output regions output by the processing stage(s) compatible with the display controller compression scheme, so that it is directly (potentially ), bypassing the display controller decompression/compression stages as appropriate (as described above).

However, it may be the case that the shape of the regions output (actuated) by one or more of the processing stages differs from the shape of the regions (windows) that the display controller compression method is configured to act on (compress). (That is, the compressed regions output by at least one of the one or more processing stages may have a different shape than the compressed regions of the output surface that the display controller presents to the display).

For example, one or more of the processing stages may include areas (i.e., height and It may be configured to output (and operate on, e.g. compress) areas whose width is each greater than one sampling position, but the display controller compression scheme is a raster line on the output side, i.e. one sampling position high and many pixels. These sampling positions may be configured to actuate (compress) rows of wide (long) sampling positions.

In one embodiment, to address this, a processing stage lossy compression scheme is used to compress together sampling locations from multiple output regions (eg tiles), e.g. preferably to different output regions (eg tiles). For example, to form a line (row) of compressed sampling positions formed by connecting each row of sampling positions from tiles.

Thus, for example, one or more of the processing stages is configured to output (operate) with regions in the form of two-dimensional blocks (arrays) of sampling locations (ie "tiles"), display If the controller compression scheme is configured to operate (compress) raster lines, i.e. rows of sampling positions together, rows of sampling positions that are compressed together using a lossy compression scheme by the processing stage in question are may be taken (preferably taken) from a plurality of blocks (tiles) of the surface output by

In a preferred such embodiment, the sampling positions from each corresponding row of each of the plurality of output surface regions (tiles) are compressed together. For example, sampling positions within each top row of tiles within a row of tiles may be compressed together. The sampling positions within each second row of tiles within this row of tiles may be compressed together, and the like.

Thus, in these embodiments, the sampling positions for each region for which the lossy compression scheme is configured to compress together may be taken (preferably) from a plurality of regions (blocks) output by the processing stage in question. is taken). Similarly, the sampling regions for each region that the processing stage lossy compression scheme is configured to compress together need not (and preferably do not) have all of the sampling locations from a given output region.

Thus, in a preferred embodiment, the processing stage compression scheme is configured to generate lossy compressed regions having a shape (using a compression window) that corresponds to the shape of the regions that the display controller compression scheme is configured to operate on (compress). (However, the shape of the regions initially output by the processing stage(s) (e.g., one or more of the frame generators) need not (and in one embodiment do not) have the same corresponding shape. ). This again makes the compressed output regions from the processing stage (potentially) compatible with the display controller compression scheme, so that they can (potentially) bypass the display controller decompression/compression stages appropriately (as described above). there will be

In these embodiments, the compressed output regions generated by the processing stage have a shape corresponding to the shape of the regions that the display controller compression scheme is configured to operate on and generate (eg, raster lines on the output surface). Rather, the size of the regions that the processing stage compression scheme produces as compressed output regions is the same as the size of the regions that the display controller compression scheme operates on (e.g., preferably, the sampling location being compressed for any given specific region). Regarding the number of ) is preferably set. This again makes the compressed output regions from the processing stage (potentially) compatible with the display controller compression scheme.

However, other configurations may also or instead be desired, such as a compressed output area generated by a processing stage compression scheme having a different size (eg number of sampling locations) than the regions the display controller compression scheme is configured to operate on. It will be possible.

As can be seen, it may be the case that the output regions (eg tiles) constituting the face are not created simultaneously (which is the case in the embodiment). This may be the case for tile-based systems, for example.

Where sampling locations from multiple output surface regions are compressed together, lossy compression may be performed once all of the relevant regions (eg tiles) have been generated. Otherwise, lossy compression may (preferably) be performed "on-the-fly", i.e., while regions (tiles) are being generated. That is, as each output region (tile) occurs, its associated sampling locations (e.g., sampling location row) are replaced by any previously compressed sampling locations (e.g., sampling location row) from previously occurring output regions (tiles). For example, corresponding sampling position rows) are preferably compressed together. In this latter embodiment, any state information necessary for the compression operation is preferably saved so that later regions (tiles) are properly compressed together with the already compressed sampling positions, e.g. carried over from tile to tile. .

In another preferred embodiment, the processing stage lossy compression scheme(s) do not (necessarily) compress regions having the shape of the regions that the display controller compression scheme is configured to operate on (compress). For example, a processing stage lossy compression scheme may select sampling locations (eg eg, a tile or tiles) may serve to compress sampling positions for a given two-dimensional block of tiles), and preferably serves.

In this case, the processing stage lossy compression scheme can be configured for the regions to be compressed, and can be block-based, for example, but in a preferred embodiment, the processing stage lossy compression scheme is a linear compression scheme, whereas Rather than compressing the rows of the output plane, it linearly compresses the sampling position data for the two-dimensional blocks of the output plane in turn.

By using a linear compression scheme, some compatibility with display controller compression schemes (i.e., if linear) can be maintained, but for example in individual blocks of the output surface generated by the processing stage rather than raster lines. By applying a linear approach, compatibility with the desired format of the output surface for processing stage operations can also be maintained.

Thus, this means that if the size and/or shape of the regions to be output (acted on) by one or more of the processing stages is not the same as the size and/or shape of the regions the display controller compression method is configured to act on (compress). It may be particularly useful.

Thus, in a particularly preferred embodiment, the processing stage lossy compression scheme uses a linear lossy compression scheme, preferably Display Stream Compression (DSC), but for linear regions of sampling locations on the output side, rather than for linear regions of the problem. It operates to linearly compress sampling location values (data values) for two-dimensional arrays (blocks) of sampling locations (eg, preferably tiles) on the output surface.

This may be done in any way if desired. For example, in one particularly preferred embodiment, where each output region comprises a two-dimensional block (array) of sampling positions (ie tiles), the plurality of rows of sampling positions of each output region are sequentially , e.g. row by row (i.e. 1st row, 2nd row, 3rd row, etc.) are preferably compressed together.

Thus, in one embodiment, each area of the output surface that the display controller provides to the display has only one row of sampling positions of the output surface; Each lossy compressed region output by at least one of the one or more processing stages has a plurality of rows of sampling positions, and the plurality of rows of sampling positions of each compressed region are in turn linearly compressed together.

The sampling locations may, of course, be compressed together using any suitable desired order or sequence. For example, in various embodiments, sampling locations may be compressed together along a Hilbert curve, or using a z-order (Morton Order), or the like.

As can be seen, regions compressed in this way may not be directly compatible with the display controller compression scheme, so it may not be possible to bypass the display controller decompression/compression stages (as described above).

Nonetheless, compressing processing stage output regions in this way is particularly useful and convenient because the integrity of each region (tile) can be maintained throughout the processing operation, which can be advantageous, for example, in tile-based data processing systems. However, since the noise characteristics (visual artifacts) generated for the compressed region will tend to be similar to the noise characteristics (visual artifacts) generated by the display controller lossy compression scheme (e.g., DSC), i.e. However, the quality of the displayed picture must not be adversely affected.

In these schemes, the processing stage compression scheme can compress sampling location blocks (tiles) that the processing stage in question individually creates (in one preferred embodiment, this has already been done). This will help maintain compatibility with the operation of the processing stage or stages in question.

Correspondingly, each set of sampling locations to be compressed may result from only one output region of a processing stage (e.g., a tile output by a processing stage) (e.g., a window of a linear compression scheme). filter) size dependent, e.g. all or part of the area of the processing stage), or by processing stage (e.g. if the compression scheme window is larger than only one output area (tile)). It may also be formed from two or more output areas (eg, tiles).

In these configurations, the sampling location blocks (regions) that are linearly compressed depend on the size of the compressed output surface regions that the display controller compression method creates (operates) (e.g., the sampling locations that are compressed as "blocks"). Although the size may be different (in terms of number), in a preferred embodiment, the size of the areas (blocks) to be compressed will be the same as the size of the areas that the display controller compression method creates (compresses). This will help enable compatibility with the display controller compression scheme.

For example, it is contemplated that the above schemes of compressing block-based output using linear compression schemes to enable compatibility with later linear compression schemes in data processing systems may themselves be novel and advantageous.

Thus, according to a further aspect of the present invention there is provided a method for linearly compressing an output data array that is generated as a plurality of two-dimensional blocks of data values that together make up the output data array, the method comprising:

forming a row of data values of an output array that is compressed using a linear compression method by concatenating data values from a plurality of data location rows from one or more two-dimensional data value blocks constituting the output data array; ,

and linearly compressing the data values of the rows thus formed to provide a linearly compressed output block of data values for the output data array.

According to a further aspect of the invention there is provided a processing stage of a data processing system comprising:

generating the output data array as a plurality of two-dimensional blocks of data values that together constitute the output data array;

forming rows of data values of an output array that are compressed using a linear compression scheme by concatenating data values from a plurality of data location rows from one or more two-dimensional data value blocks constituting the output data array;

and processing circuitry configured to linearly compress the rows of data values thus formed to provide a linearly compressed output block of data values for the output data array.

As will be appreciated by those skilled in the art, these aspects of the present invention may, and preferably include, as appropriate, one or more or all of the preferred optional features of the present invention described herein.

Thus, for example, the output data array is preferably an output surface that is used as, or to form, an output surface for display, and the two-dimensional blocks of data values are formed by a processing stage generating the output surface. Preferably it is each processing tile that acts and creates as its output.

Correspondingly, the processor preferably repeats the above process for all blocks comprising the output data array to provide a linearly compressed version of the output data array.

The order in which the data values from the plurality of data location rows are concatenated can be selected as desired. In a particularly preferred embodiment, the data values from the plurality of data location rows are of Hilbert order; Z-order; and data value block row units.

In one embodiment, a row of data values that is compressed using a linear compression scheme is formed by (at least) concatenating data values from a plurality of data location rows from one data value block constituting an output array of data values. eg by connecting a first row of blocks, a second row of blocks, and a third row of blocks, as appropriate).

Correspondingly, in another embodiment, a row of data values that is compressed using a linear compression scheme is obtained by concatenating data values from data location rows from a plurality of other data value blocks that make up the output data array. , preferably by concatenating each corresponding data location row (e.g., the n-th row) from each of a plurality of data value blocks of a row of data value blocks constituting the output array (e.g., preferably concatenates a first data location row from each of the data value blocks in the row of data value blocks to form one row of data values that are linearly compressed, and forms a second row of data values that are linearly compressed. by concatenating second row data values from each of the data value blocks in the row of data value blocks, etc.).

It may also be possible to use other types of lossy compression schemes that are based on and/or configured to match (eg, in terms of their characteristics) the display controller lossy compression schemes.

In this case (or not), the processing stage compression scheme may be “fitted” to the display controller lossy compression scheme in any suitable and desired manner. For example, one or more of the least significant bits of the output of a given compression scheme may be discarded in order to conform it to the display controller compression scheme, resulting in format loss. Alternatively or additionally, the compression ratio (rate controller) of the processing stage lossy compression scheme may be controlled based on the display controller compression scheme.

For example, in one embodiment, a lossy compression scheme may include a version of ARM Frame Buffer Compression (AFBC) that is lossy, for example by discarding one or more of the least significant bits.

In a preferred embodiment, the display controller is directed to one or more of the processing stage(s) providing a compressed output (e.g., preferably the display controller is associated with its lossy compressed output that it is providing to the display). It is operated to send (feedback) information. This information is then preferably used to control (ie, adjust) the lossy compression scheme of the processing stage(s) in question. As such, the processing stage lossy compression scheme is preferably controlled based on information (feedback) related to the (currently in use) display controller lossy compression scheme.

In one preferred embodiment, the feedback information includes information about the characteristics, preferably loss, of the compression of the display controller, and is preferably used to control the compression ratio (rate controller) of the processing stage lossy compression scheme. . For example, in one embodiment, the display controller compression ratio for a particular (eg, current and/or previous) face or faces (frame or frames) is, for example, the next face or faces (frame or frames). ), can be used to control the compression ratio of the processing stage lossy compression scheme. This can ensure that the processing stage lossy compression scheme properly “matches” the display controller compression scheme.

The processing stages and display controller, etc. may process (eg, generate and/or otherwise process) their respective faces as a whole (eg, as a full frame). In this case, the output surface area will contain the entire frame. However, in a preferred embodiment, at least one, preferably all of the processing stages and/or the display controller have their respective sub-regions (sub-areas) of the entire surface (eg frame) in question. treat the side

In this case, the regions of the faces considered and used will each include a portion (some but not all) of (the area of) the face in question (ie the face or faces are divided into a plurality of regions). The areas of these faces may each represent any suitable desired area (area) of the face in question. Subdivision of faces into regions is as desired, as long as it is possible for the face in question to be subdivided or divided into a plurality of small identifiable regions, each representing a portion of the overall face that can be identified and processed in the manner of the present invention. can be done

In a preferred embodiment, regions correspond to each block of data corresponding to each portion of the overall array of data representing the face in question (as is known in the art, faces are usually sampling locations (e.g., pixels). ) will be represented as arrays of data, and stored as arrays).

All faces can be divided into regions of the same size and shape (in one preferred embodiment this is done), or else the different faces can be divided into regions and regions of different sizes.

Each face region (eg, block of data) in one embodiment may represent a different portion (region) of the face (the entire array of data) (although the regions may overlap if desired). Each area (data block) should ideally represent a suitable portion (area) of a surface (data array), such as a plurality of data locations (sampling locations) within the surface. A suitable region size may be, for example, 8x8, 16x16 or 32x32 data locations in a plane data array.

In some embodiments, these faces are divided into regions (eg, blocks of data) of irregular size and shape, preferably in the form of squares or rectangles. However, this is not required and other methods may be used if desired.

In some embodiments, each face area corresponds to a processing tile that a graphics processor or video engine or composition stage or camera ISP or the like that renders, generates, or otherwise provides a face creates as its output. This is a particularly simple way of realizing the present invention, as the graphics processor, display engine or controller will directly generate the tiles, so that no additional processing will be required to "create" the surface regions considered in the present method.

(As is known in the art, for example in tile-based rendering, the two-dimensional output array or frame of the rendering process (e.g., normally displayed to display the scene being rendered) A target") is, for the purpose of the rendering process, subdivided or divided into a plurality of small areas, usually called "tiles". Each of these tiles (areas) is rendered separately (usually in turn). These rendered tiles The (regions) then form a complete output array (frame) (render target), eg for display.

Other terms commonly used for "tiling" and "tile-based" rendering include "chunking" (these areas are called "chunks") and "bucket" rendering . The terms "tile" and "tiling" will be used herein for convenience, but it should be understood that these terms are intended to encompass all alternatives and like terms and techniques.

In these configurations of the present invention, the tiles from which faces are divided for purposes such as rendering and/or composition may be of any desired suitable size or shape. The tiles in some embodiments are, but need not be, all of the same size and shape, as is known in the art. In some embodiments, each tile is rectangular (including square), preferably 8x8, 16x16, 32x32, 32x4 or 32x1 sampling positions. Areas that are not square rectangles, such as 32x4 or 32x1, may be more suitable for output to a display.

In some embodiments, the present invention may be performed, preferably also or instead, using surface regions of different sizes and/or shapes for the tiles on which a process, such as rendering, operates (generates).

For example, in some embodiments, surface regions contemplated in the manner of the present invention may consist of a set of multiple “treatment” tiles, and/or may consist of only sub-portions of treatment tiles. In these cases, there may be an intermediate stage that actually "generates" the desired face regions from the "process" tile or tiles in question.

In these configurations, when one side is treated as being composed of a plurality of areas, the technique of the present invention should and is preferably used for a plurality of, preferably each area of the side. Thus, in a preferred embodiment, a plurality of areas, preferably each area, of a face are treated with the method of the present invention. In this way, the entire output surface (eg) will be generated by the process of the present invention.

Although the present invention has been described above with particular reference to the processing of a given display output surface, as will be appreciated by those skilled in the art, the present invention is intended to provide a plurality of output surfaces for display, preferably display on a display. can be used, and is preferably used, to provide a sequence of output faces (eg frames) that are

The various stages of a data processing system may be, for example, one or more fixed (i.e., dedicated to one or more functions that cannot be changed), e.g., by programmable circuitry that can be programmed to perform desired operations. It may be implemented as desired in the form of a functional unit (hardware) or as one or more programmable processing stages. There may be fixed functions and programmable stages.

One or more of the various processing stages of the present invention may be provided as discrete circuit elements. Additionally or alternatively, some or all of the stages may be formed at least in part from shared circuitry.

One or more of the various stages of the present invention may be operated to perform its function at all times for certain receiving surfaces and for all receiving surfaces. Additionally or alternatively, one or more stages may be operated to selectively perform their function with respect to the receiving surfaces, ie when desired and/or appropriate.

The data processing system also includes one or more, preferably all of, a central processing unit, a graphics processing unit, a video processor (codec), a system bus, a memory controller, and additional components, as known to those skilled in the art. It can be done, and it is preferably provided.

The data processing system may include external memory (eg, via a memory controller), one or more local displays, and/or (the invention extends to a configuration that includes one or more) one or more external displays. It may be, and is preferably configured, to communicate with more than one.

In one embodiment, the data processing system further includes a display or the display. The display on which the display controller is used may be any suitable desired display, such as, for example, a screen or a printer.

The invention may be implemented in any suitable system, such as an appropriately configured microprocessor based system. In one embodiment, the invention is implemented in a computer and/or microprocessor based system.

The various functions of the present invention may be performed in any desired and suitable manner. For example, the functions of the present invention may be implemented in hardware or software, as desired. As such, for example, and although not expressly stated otherwise, the various functional elements and “means” of the present invention, such as suitable dedicated hardware elements and/or programmable hardware elements that can be programmed to operate in a desired manner, It may have a suitable processor or processors, controller or controllers, functional units, circuitry, processing logic, configuration as a microprocessor, etc., operable to perform various functions and the like.

It should be noted herein that, as will be appreciated by those skilled in the art, various functions and the like of the present invention may be replicated and/or performed in parallel on a given processor. Likewise, the various processing stages may share processing circuitry, etc., if desired.

Depending on what hardware is required to perform the specific functions described above, the graphics processing pipeline may also include any one, more, or all of the normal functional units that the graphics processing pipeline includes.

In addition, it will be appreciated by those skilled in the art that all of the described embodiments of the present invention may suitably include, and in one embodiment include, any one or more or all of the features described herein. will be.

The methods according to the invention may be implemented at least partly using software, for example computer programs. In this further embodiment, the present invention provides computer software that is particularly suitable for performing the methods described herein when installed on a data processor, and for performing the methods described herein when program elements are executed on the data processor. It will be appreciated that a computer program element consisting of computer software code portions to do so, and a computer program comprising code suitable for performing all the steps of the method or methods described herein when the program is executed on a data processing system. . The data processor may be a microprocessor system, a programmable field programmable gate array (FPGA), or the like.

The present invention also relates to a microprocessor system having a data processor, a computer configured with a data processor when used to operate a renderer or graphics processor, software that causes the processor, renderer or system to perform the steps of the methods of the present invention. It also extends to computer software carriers. Such a computer software carrier may be a physical storage medium such as a ROM chip, CD ROM, RAM, flash memory, or disk, or may be a signal such as an electronic signal using a wire, an optical signal, or a radio signal such as a satellite.

Since not all steps of the methods of the present invention need be performed by computer software, in a broader embodiment the present invention is directed to computer software installed on a computer software carrier for performing at least one of the steps of the methods described herein; and You may find out more about providing such software.

The invention may accordingly be suitably implemented as a computer program product for use in a computer system. Such an implementation may include a set of computer readable instructions fixed on a tangible, non-transitory medium, such as a diskette, CD ROM, ROM, RAM, flash memory, or hard disk. It also uses intangible wireless technology, including but not limited to microwave, infrared or other transmission technology, through a modem or other interface device, through a tangible medium, including but not limited to optical or analog communication lines, or Thus, it may include a series of computer readable instructions transmittable to a computer system. A series of computer readable instructions implements all or some of the functionality previously described herein.

Those skilled in the art will appreciate that such computer readable instructions may be written in a number of programming languages for use with many computer architectures or operating systems. Further, such instructions may be stored using any memory technology, present or future, including but not limited to semiconductor, magnetic, or optical, or optical, infrared, or microwave. It may be transmitted using any communication technology, present or future, including but not limited to. Such a computer program product may be distributed as a removable medium to which printed or electronic documents are affixed, for example shrink wrapped software, or may be preloaded onto a computer system, for example a system ROM or fixed disk. or may be distributed from a server or bulletin board via a network, for example, the Internet or the World Wide Web.

Hereinafter, various embodiments of the present invention will be described as an example with reference to the accompanying drawings.

1 schematically illustrates a data processing system according to an embodiment of the present invention.
2 is a schematic representation of various data processing system processes.
3 schematically illustrates a data processing system process according to one embodiment of the present invention.
4 illustrates a display controller process according to one embodiment of the present invention.
5 schematically illustrates a compression scheme according to embodiments of the present invention.
6 schematically illustrates a data processing system process according to one embodiment of the present invention.

Like reference numbers are appropriately used for like elements throughout the drawings.

1 will schematically show a data processing system 1 . The data processing system 1 includes a camera image signal processor (ISP) 2, a central processing unit (CPU) 3, a graphics processing unit (GPU) 4, a video engine or codec 5, (eg , an image processor 6 (which may also include a composition engine), a display controller 7, and a memory controller 8. As shown in FIG. 1, they communicate via interconnect 9 and access off-chip main memory 10. The camera ISP 2 communicates with the camera's image sensor 11 to receive image data from the camera. Display controller 7 communicates with display 12 via a display interface so that output frames (faces) are displayed on display 12 .

Figure 2a shows a method of operating the data processing system 1 without compression. A frame generator 20 (which may include, for example, a camera ISP 2, a CPU 3, a GPU 4, a video engine 5, or an image processor 6) creates frames (planes). generated and stored in the main memory 10, for example a frame buffer. The stored frame is read from memory 10 by display controller 7, which sends the frame to display 12 for display.

Writing to and reading from the frame buffer and sending pixel data to the display requires a significant amount of bandwidth. For example, reading one high-definition (HD) frame consumes approximately (1920×1080×24 bpp×60 Hz) 356 MB/s. One ultra-high definition (4K) frame fetch consumes approximately (3840×2160×24 bpp×60 Hz) 1.4 GB/s. These bandwidth numbers are actually increased because the data is all read from memory (e.g. by display controller 7 and/or image processor or composition engine 6, etc.) and into memory (e.g. by Because they need to be written (by the GPU 4 and/or the video decoder 5), there may be multiple processing stages in a data processing system, and multiple frames are the final frame (eg when using composition). may need to be processed to generate

2B shows a method of operating the data processing system 1 that compresses pixel data sent from the display controller 7 to the display 12 using, for example, Display Stream Compression (DSC).

DSC is a lossy compression standard designed to provide a known fixed bit rate and be computationally inexpensive with low latency. The algorithm uses Delta Pulse Code Modulation (DPCM), Indexed Color History (ICH), and works on raster lines. A rate controller is used to control compression loss. DSC can compress data by approximately 66% in the display interface (so 24 bpp becomes 8 bpp for example).

As in the configuration of FIG. 2A, the frame generator 20 in FIG. 2B generates a frame (plane) and stores it in the main memory 10, for example, a frame buffer, and then the stored frame is stored in the display controller ( 7) to read from the memory 10. The display controller 7 then compresses the frame (plane) using a compression stage 21 before sending the frame to the display 12 for display. Compressing the display controller output in this way can reduce the overall bandwidth and power requirements of the data processing system.

2C shows a method of operating the data processing system 1 that compresses the frames (faces) generated by the frame generator 20. In this embodiment, frame generator 20 generates frames (planes) that are compressed by compression stage 22. The compressed frame (plane) is stored in the main memory 10, for example in a frame buffer, and then read by the display controller 7. As shown in FIG. 2C , in this embodiment, the compressed frame may be decompressed in the display controller 7 using the decompression stage 23 . This may be the case, for example, if the compression method used to compress the frame is not the same (or not similar enough) to the display controller compression method and/or the frame (face) requires modification by the display controller 7. It may be suitable for the case of .

The display controller 7 uses a compression stage 21 (for example, using Display Stream Compression (DSC)) before sending the decompressed frames to the display 12 for display (optionally modified). Compress the decompressed frame.

Compressing both the generated frame and the display controller output in this way further reduces the overall bandwidth and power requirements of the data processing system.

Applicants find that when a display output is compressed using lossy compression (DSC, etc.), using the same lossy compression scheme, or using another lossy compression scheme with similar noise characteristics, the quality lost by upstream compression is lost. It has been recognized that compressing data (e.g., generated frames) further up the pipeline will not adversely affect the quality of the displayed picture, since this will be lost by display output compression under any circumstances.

Figure 2d shows another method of operating the data processing system 1 that may compress the frames generated by the frame generator 20 using a compression scheme identical or similar to the display controller compression scheme. The frame generator 20 generates frames (planes) that are compressed by the compression stage 22, for example using Display Stream Compression (DSC). The compressed frame is stored in the main memory 10, for example in a frame buffer. The stored compressed frame is then read from memory 10 by display controller 7 . In this embodiment, since the compressed frame has already been compressed using the display controller compression scheme, it does not need to be decompressed by the decompression stage 23, nor does it need to be compressed using the compression stage 21. There is no , and is instead configured to bypass the decompression stage 23 and the compression stage 21 . The compressed frame is then passed to the display 12 for display.

In this embodiment, the overall bandwidth and power requirements of the data processing system are reduced in the same way as the configuration of FIG. 2C. Furthermore, the processing requirements of the data processing system can be further reduced since the decompression stage 23 and the compression stage 21 in the display controller 7 can be bypassed when appropriate.

The data processing system 1 of the present embodiment optionally includes, for example, a specific frame generator (e.g., a camera ISP 2, a CPU 3, a GPU 4, a video engine 5, or an image processor 6). )) and/or as appropriate to perform any or all of the methods of FIGS. 2A-2D .

Although the above embodiments have been described with respect to the display controller 7 for displaying the frame and the frame generator 20 for generating the frame, in various embodiments, the frame (surface) generated by the frame generator is, for example, Before this generated frame or a frame (surface) generated using it is displayed, it goes through one or more additional steps of processing.

Figure 3 shows one such embodiment in which the composition engine 6 generates composited frames (faces). In the embodiment of Figure 3, the camera ISP 2, GPU 4 and video decoder 5 generate one or more faces (frames). As shown in FIG. 3, one or more of the one or more generated faces may be compressed before being stored, for example, in memory 10 (not shown).

Composition engine 6 then reads one or more of the generated faces from, for example, memory 10 . In this embodiment, the composition engine 6 may decompress those faces that are compressed before compositing the faces to generate a synthesized output frame (face).

The synthesized output frame may then be compressed before being stored in main memory 10 and then read from memory 10 by display controller 7 . The display controller 7 may, for example, if the compression method used to compress the frame is not the same (not similar enough) as the display controller compression method, and/or if the frame is added by the display controller 7 If changes are required, the read frame can be decompressed. Once the frames are decompressed, the display controller 7 may then compress the frames before sending them to the display 12 for display, for example using Display Stream Compression (DSC).

If the compression method used to compress the frame is the same as (sufficiently similar to) the display controller compression method, and if the frame does not require further modification by the display controller 7, the display controller 7 may, for example, , decompression and compression stages can be bypassed to send the read compressed frame directly to the display for display.

The frame (surface) is an image enhancement method; rotation and/or scaling; and/or one or more other steps in processing, such as changing contrast and/or brightness, performing gamma correction, and/or performing tone mapping, as desired, to Level Adaptive Overdrive (LAO). ), FFD (Feed Forward Drive), DCC (Dynamic Contrast Compensation), etc., display responsiveness may be improved. Such processing may be performed by the display controller 7 or in one or more stages as appropriate, for example prior to compression.

While the above embodiments have been described with respect to the display controller 7 passing an output frame to the display 12 for display, in further embodiments, in addition or instead, the display controller output may be passed to a video encoder. One such embodiment is shown in FIG. 3 , where a display controller 7 converts output frames to a video encoder (eg for selective (eg wireless) transmission and encoding to another display). 30) can be selectively passed on. A video encoder can decompress frames, encode decompressed frames, and then compress encoded frames, as desired.

Additionally or alternatively, display controller 7 may write compressed frames to memory 10 .

Although the above embodiments have been described in terms of a data processing system that operates on entire frames (planes), in other embodiments the data processing system of the present invention is directed to, for example, the tiles or blocks of data that make up each frame. For , actuate the parts of each frame. Thus, in this embodiment, frame generator 20 is operative to generate and write one or more portions of one or more frames to memory 10, and display controller 7 may write one or more such portions to memory 10. (10), compression 21, 22 and decompression 23 stages are operative to compress or decompress one or more such portions.

In this embodiment, the data processing system may be operable to perform any or all of the methods of FIGS. 2A-2D, for example, optionally depending on the particular frame (plane) portion in question.

4 shows such a method of operating the display controller 7 . Each new picture (input plane (frame)) to be displayed (which may be generated, for example, by the ISP 2, CPU 3, GPU 4, video engine 5 or image processor 6) ), the display controller 7 fetches (reads) a block of data corresponding to one part (area) of the frame (surface) in step 41. In this example, it is assumed that each data block is a processing tile of the face (frame) in question, but other configurations are of course possible.

In step 42, it is determined whether the block is compressed. If the block is not compressed, then the display controller preferably manipulates or changes the block if necessary at step 45, then compresses the block using DSC at step 50 and outputs it at step 52.

If the block is compressed, step 43 determines whether the block is compressed using DSC. If the block is compressed using a compression method other than DSC, the block is decompressed in step 44, the block is manipulated or altered if necessary in step 45, the block is compressed using DSC in step 50, and the block is displayed in step 52. A block is output for

If a block is compressed using DSC, step 46 determines whether the block requires manipulation or alteration. If the block does not require manipulation or alteration, in step 51 the display controller 7 presents the block to the display.

If the block requires manipulation or modification, in step 47 the display controller 7 decompresses the block, and in step 48 the block is manipulated or altered as necessary.

In step 49, it is determined whether the manipulated or altered block is the same or sufficiently similar to the original block. It may be the case that the difference between the modified block and the original block is small enough, eg below a threshold, that when the block is displayed, no difference will be noticed.

If the manipulated or altered block is identical or sufficiently similar to the original block, then in step 51 the display controller 7 presents the original compressed block to the display. Manipulated or altered blocks are discarded.

If the manipulated or altered block is not identical or sufficiently similar to the original block, at step 50 the display controller compresses the altered block using DSC and at step 52 outputs the block for display.

In this way, when a block is changed, a changed version of the block or an unaltered version of the block is output by the display controller 7, depending on whether this change caused sufficient change in the block. If the unchanged version of the block is output by the display controller 7, the changed version of the block is discarded and not compressed by the display controller 7, thereby reducing the throughput required by the display controller 7. do.

In one alternative embodiment, no check is made at step 49, and the manipulated or changed block is compressed and output for display.

As shown in Fig. 4, this process is repeated for each block of the frame (surface) until the frame has been completely output for display (at step 55).

As described above, in this embodiment, DSC, or another lossy compression method having similar noise characteristics to DSC, is used to compress upstream data (e.g., a generated frame).

If the upstream compression scheme is DSC, the processing stage that generates the frame can be operated to output each frame as a plurality of raster lines (ie rows) of pixels. Each such generated raster line can then be compressed as usual using DSC.

Alternatively, if the processing stage generating the frame outputs each frame as a plurality of tiles, then the pixels in each top row of tiles within each row of tiles may be compressed together, and the row of tiles The pixels in each second row of tiles within may be compressed together. As can be seen, each of the tiles that make up a face will generally occur in sequence, ie not simultaneously. Thus, in this embodiment, state information is carried between tiles to assist in the compression process while the tiles are being generated.

5A shows such a linear compression scheme that operates to compress each raster line in each frame, i.e. pixels A1-A15, and pixels B1-B15.

This compression scheme ensures that the compressed regions will be directly compatible with the display controller's DSC compression scheme, and can therefore be configured to bypass the display controller decompression/compression stages when appropriate (as described above).

In another embodiment, the lossy compression scheme is configured to compress each tile separately by compressing each of the multiple rows of data locations in the tile in order, namely the first row, the second row, the third row, etc. together. .

This is shown in Figure 5b. Each tile (consisting of a 3x3 group of pixels in Figure 5b) is compressed together using the DSC algorithm. As shown in Fig. 5b, this is done by rearranging the pixels A1-A15 to be compressed together.

Tiles compressed in this way will not be able to bypass the display controller decompression/compression stages, but this way the noise characteristics introduced by compression are very close to those introduced by normal linear DSC compression. while ensuring compatibility with tile-based systems.

Other pixel arrangement schemes may also be used, for example along the Hilbert curve, or in z-order.

Raster line and tile based compression schemes can be implemented using similar hardware, so this hardware can be reused.

The compression algorithm used, compression block shape (eg, tile or line), and compression ratio may be selected based on display output compression. For example, if blocks are expected to be manipulated, a tile-based compression scheme may be more desirable than a line-based scheme. Line-based DSC compression can be used once the frame generator has generated horizontal lines of tiles, i.e. the compression state can be followed over the next tile being processed.

The rate controller of the compression scheme can be adjusted so that the compression scheme has properties similar to raster scan DSC and/or can minimize visible compression artifacts based on the size of the tiles.

In an alternative embodiment, other compression schemes may be implemented that have compression artifacts similar to raster scan DSC. For example, a lossy compression scheme may have a modified version of ARM Frame Buffer Compression (AFBC) which is made lossy by discarding one or more of the least significant bits.

Other compression schemes may of course also be used.

Figure 6 shows an embodiment in which upstream compression can be tuned. Frame generator 20 (e.g., ISP 2, GPU 4, video decoder 5, or image processor 6) generates compressed frames. After the frame is written to the memory 10, it is passed to the display controller 7. In this embodiment, the display controller 7 can perform some processing on frames. For example, the frame may be decompressed, then compressed using a lossy compression scheme, and then passed to the display 12.

In this embodiment, the display controller 7 provides the compression adjustment information back to the frame generator 20. This can be used to tune upstream compression. For example, lossy raster scan compression can be fed back to an upstream lossy compression stage and used to properly control the compression ratio. The display controller compression ratio for the current and/or previous frame or frames may be used to control the compression ratio of the processing stage lossy compression scheme, for example, for the next frame or set of frames.

In other words, the frame generator 20 can generate its data with appropriate precision depending on the adjustment information provided to it by the display controller 7. For example, frame generator 20 may generate lower quality frames if it knows that display controller 7 lossy compression will discard more than a specified amount of data.

In these embodiments, there may be multiple processing stages. For example, frame generator 20 may pass the frame to memory 10, which is read by composition engine 6, which composites multiple frames to generate a composited frame. After the synthesized output frame is written to the memory 10, it is read and displayed by the display controller 7.

In one embodiment, where the camera ISP has a direct interface with the display controller 7, the output from the camera ISP may be compressed using DSC compression. This reduces power consumption by reducing the amount of data passed from the ISP to the display controller.

It can be seen from the foregoing that the present invention provides an improved data processing system that requires a reduced amount of memory bandwidth and power without adversely affecting the quality of the displayed picture. This is achieved in a preferred embodiment by lossy compressing at least one face used to generate the output face for display upstream of a display controller providing the output face to the display in lossy compressed form.

Claims (25)

As a data processing system,
a display controller operative to provide a compressed version of an output surface to be displayed to a display that has been compressed using a lossy compression scheme;
having one or more processing stages operative to provide a surface or surfaces to be used in forming an output surface to be displayed;
At least one of the one or more processing stages is operative to output the surface or one or more regions of surfaces, provided by the processing stage, used in forming the output surface to be displayed in compressed form using a lossy compression scheme. become,
the display controller includes a compression stage and a decompression stage;
The data processing system is configured such that one or more regions output by at least one of the one or more processing stages in a compressed form using a lossy compression method bypass the display controller compression stage and decompression stage. Characterized in that, the data processing system.
According to claim 1,
A data processing system, characterized in that the lossy compressed output surface provided to the display comprises one or more areas that have been compressed by the display controller and/or one or more areas that have been compressed by other than the display controller.
According to claim 1 or 2,
The system,
one or more frame generator processing stages operative to output one or more regions of one or more faces that have been compressed using a lossy compression scheme;
a composition processing stage operative to output one or more regions of the synthesized output surface that have been compressed using a lossy compression scheme;
of one or more compression processing stages operative to output one or more regions of one or more faces that have been compressed using a lossy compression scheme.
A data processing system characterized by having one or more or all of them.
According to claim 1 or 2,
A data processing system according to claim 1, wherein the lossy compression scheme used by at least one of the one or more processing stages is based on a display controller lossy compression scheme.
According to claim 1 or 2,
Noise characteristics of a lossy compression scheme used by at least one of the one or more processing stages match noise characteristics of a display controller lossy compression scheme.
According to claim 1 or 2,
The size and/or shape of each of the compressed regions output by at least one of the one or more processing stages is determined by the size and/or shape of each region of the plurality of regions of the output surface provided to the display by the display controller. A data processing system characterized in that it corresponds to a shape.
According to claim 1 or 2,
The size and/or shape of each of the compressed regions output by at least one of the one or more processing stages is determined by the size and/or shape of each region of the plurality of regions of the output surface provided to the display by the display controller. A data processing system characterized by being different from the shape.
According to claim 1 or 2,
At least one of the one or more processing stages comprises:
generating an output data array as a plurality of two-dimensional blocks of data values that together constitute the output data array;
forming rows of data values of an output array to be compressed using a lossy compression scheme by concatenating data values from a plurality of data location rows from one or more two-dimensional data value blocks constituting the output data array;
characterized in that it is operative to linearly compress rows of data values so formed using a lossy compression scheme to provide a compressed output area for said output data array.
According to claim 8,
The data values from the plurality of data location rows are of Hilbert order; Z-orer; and a data value block row-by-row order.
According to claim 1 or 2,
A data processing system, characterized in that a lossy compression scheme used by at least one of said one or more processing stages is controlled based on display controller lossy compression.
delete delete A display controller operable to read a face or faces from memory and provide a compressed version of an output face to be displayed to a display that has been compressed using a lossy compression scheme, and a face or faces used in forming the output face to be displayed. A method of operation of a data processing system having one or more processing stages operable to provide, said method of operation comprising:
one or more of said one or more processing stages providing a surface or surfaces used in forming an output surface to be displayed;
reading at least one of the faces presented by the processing stage from memory, using this face to form an output face to be displayed, and providing the output face to a display for display in compressed form using a lossy compression scheme; having the display controller;
at least one of the one or more processing stages outputs a surface or one or more regions of surfaces, provided by the processing stage, used in forming an output surface to be displayed in compressed form using a lossy compression scheme;
the display controller includes a compression stage and a decompression stage;
The method of operation comprises one or more regions output by at least one of the one or more processing stages in a compressed form using a lossy compression scheme that bypasses the display controller compression stage and decompression stage. to, how it works.
According to claim 13,
A method of operation, characterized in that the lossy compressed output surface provided to the display comprises one or more areas compressed by the display controller and/or one or more areas compressed by other than the display controller.
According to claim 13 or 14,
one or more frame generator processing stages output one or more regions of one or more faces that have been compressed using a lossy compression scheme; and/or
A composition processing stage outputs one or more regions of the synthesized output surface that have been compressed using a lossy compression scheme;
A method of operation, characterized in that one or more compression processing stages output one or more regions of one or more surfaces that have been compressed using a lossy compression scheme.
According to claim 13 or 14,
characterized in that the lossy compression scheme used by at least one of the one or more processing stages is based on a display controller lossy compression scheme.
According to claim 13 or 14,
characterized in that noise characteristics of a lossy compression scheme used by at least one of said one or more processing stages match noise characteristics of a display controller lossy compression scheme.
According to claim 13 or 14,
The size and/or shape of each of the compressed regions output by at least one of the one or more processing stages is determined by the size and/or shape of each region of the plurality of regions of the output surface provided to the display by the display controller. Characterized in that it corresponds to the shape, the operating method.
According to claim 13 or 14,
The size and/or shape of each of the compressed regions output by at least one of the one or more processing stages is determined by the size and/or shape of each region of the plurality of regions of the output surface provided to the display by the display controller. A method of operation, characterized in that it is different from the shape.
According to claim 13 or 14,
generating an output data array as a plurality of two-dimensional blocks of data values that together constitute the output data array;
forming rows of data values of an output array to be compressed using a lossy compression scheme by concatenating data values from a plurality of data location rows from one or more two-dimensional data value blocks constituting the output data array;
linearly compressing the rows of data values so formed using a lossy compression scheme to provide a compressed output area for the output data array.
and further comprising at least one of said one or more processing stages.
21. The method of claim 20,
wherein the data values from the plurality of data location rows are concatenated using one of Hilbert order, Z-order, and data value block row-by-row order.
According to claim 13 or 14,
and controlling a lossy compression scheme used by at least one of said one or more processing stages based on display controller lossy compression.
delete delete A computer program, stored in a medium, comprising computer software codes for performing the operation method of claim 13 or 14 when the program is executed in data processing means.

Images