GB2550942A - Display Data processing - Google Patents
Display Data processing Download PDFInfo
- Publication number
- GB2550942A GB2550942A GB1609605.9A GB201609605A GB2550942A GB 2550942 A GB2550942 A GB 2550942A GB 201609605 A GB201609605 A GB 201609605A GB 2550942 A GB2550942 A GB 2550942A
- Authority
- GB
- United Kingdom
- Prior art keywords
- display
- display data
- display control
- control device
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 claims description 52
- 230000006835 compression Effects 0.000 claims description 14
- 238000007906 compression Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 12
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
- G06F3/1431—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display using a single graphics controller
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
- G06F3/1438—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display using more than one graphics controller
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
- G09G5/363—Graphics controllers
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2350/00—Solving problems of bandwidth in display systems
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/06—Consumer Electronics Control, i.e. control of another device by a display or vice versa
Abstract
A display system for displaying display data includes a host device 11 having a graphic display unit(GPU) 14, a plurality of display control devices 12, each having their own processing unit 15 and memory 16 and a plurality of display devices 13, at least two of the display devices 13 having different characteristics. A display data processing pipeline receives an initial input of coded display data at the host device 11, includes a plurality of serially connected display data processing steps and generates a final output of rendered display data at each display control device 12 for transmittal to a respective display device 13 for display. The system involves determining the characteristics of each of the display devices 13 connected to each of the display control devices 12 to determine characteristics of the rendered display data to be transmitted from the final output of the display data processing pipeline to the particular display device 13 and determining processing capabilities of each of the display control devices 12. The host device 11 then determines a subset of the processing steps that are to be performed by the particular display control device 12, performs all the processing steps from the initial input to a processing step performed prior to the subset of processing steps to be performed by the particular display control device 12 and outputs part processed display data to the particular display control device 12 to enable the particular display control device 12 to perform the subset of processing steps ending at the final output.
Description
Display Data Processing
Background
In 3D graphical processing, there are a number of display data processing steps which are conventionally run on a computing device which has generated display data, commonly known as a host device, prior to the finished display data being transmitted to a connected display device for display. Because the host device has a limited capacity for such processing, the number of varied display devices that can be connected and supplied with data is necessarily limited.
Shaders are sets of pre-generated standard instructions which are conventionally run entirely on the host device at each step in the display data processing pipeline. Each application or other graphics producer on a host device may run its own instance of a standard shader simultaneously in a multiplexed system controlled by the Graphical Processing Unit (GPU) of the host device.
Since the shaders may handle different parts of the display data processing pipeline, they have different purposes. Vertex shaders take vertices as input and perform transformations on them such as rotation and scaling. Geometry shaders take primitives such as shapes and lines as input and modify them prior to rasterization, which converts vertices to pixels. Pixel shaders, sometimes also known as Fragment shaders, take pixel data as input and produce pixel output which has been blended, lit, etc..
Because all such processes are carried out on the host device, there is a processing bottleneck on the host device which makes it more difficult to connect multiple display devices or especially large display devices. The invention aims to mitigate this problem.
Summary of the Invention
Therefore according to first aspect, the invention provides a host device for use in a display system for displaying display data, the display system comprising a host device, a plurality of display control devices and a plurality of display devices, at least two of the display devices having different characteristics, each display control device of the plurality of display control devices being connected to the host device and to a respective display device, wherein the display data is processed in a display data processing pipeline receiving an initial input of coded display data at the host device and generating a final output of rendered display data at each display control device for transmittal to a respective display device for display, the display data processing pipeline including a plurality of serially connected display data processing steps performed on the display data, wherein the host device is configured to: determine the characteristics of each of the display devices connected to each of the display control devices to determine characteristics of the rendered display data to be transmitted from the final output of the display data processing pipeline to the particular display device; determine processing capabilities of each of the display control devices connected to the host device; determine, based on at least the determined characteristics of the rendered display data required for a particular display device and on the processing capabilities of a particular display control device connected to the particular display device, a subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device, the subset ending at the final output; perform all the display data processing steps from the initial input to a display data processing step performed prior to the subset of the plurality of serially connected display data processing steps to generate part-processed data; and outputting the part-processed display data to the particular display control device to enable the particular display control device to perform the subset of the plurality of serially connected display data processing steps ending at the final output.
Preferably, each display data processing step is performed on a corresponding display data processing engine executing a selected shader program.
In one embodiment, the host device comprises all the display data processing engines required for the display data processing pipeline.
The host device may, preferably, be configured to determine a number of display data processing engines located on the particular display control device based on the processing capabilities of the particular display control device to determine the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device.
Preferably, the host device is configured to select a shader program for each of the display data processing engines in the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device and to transmit instructions to the particular display control device to control the particular display control device to execute the selected shader program on the corresponding display data. In one embodiment, the host device is configured to transmit the selected shader program to the particular display control device if the particular display control device does not already have the selected shader program stored in memory.
According to a second aspect, the invention provides a display control device for use in a display system, the display system comprising a host device, a plurality of display control devices and a plurality of display devices, at least two of the display devices having different characteristics, each display control device of the plurality of display control devices being connected to the host device and to a respective display device, wherein the display data is processed in a display data processing pipeline receiving an initial input of coded display data at the host device and generating a final output of rendered display data at each display control device for transmittal to a respective display device for display, the display data processing pipeline including a plurality of serially connected display data processing steps performed on the display data, wherein the display control device is configured to: transmit to the host device the characteristics of the display device connected to the display control device to enable the host device to determine characteristics of the rendered display data to be transmitted from the final output of the display data processing pipeline to the particular display device; transmit to the host device processing capabilities of the display control device; receive, from the host device, information regarding a subset of the plurality of serially connected display data processing steps that are to be performed by the display control device, the subset ending at the final output; receive, from the host device, part-processed display data to enable the display control device to perform the subset of the plurality of serially connected display data processing steps; perform the display data processing steps in the subset of the plurality of serially connected display data processing steps on the part-processed display data to generate the rendered display data; and transmit the rendered display data to the display device connected to the display control device.
Preferably, each display data processing step is performed on a corresponding display data processing engine executing a selected shader program.
In one embodiment, the display control device comprises fewer than all the display data processing engines required for the display data processing pipeline.
The display control device may, preferably, be configured to receive instructions from the host device to execute a selected shader program on a corresponding display data processing engine to process the display data, the selected shader program being selected by the host device for each of the display data processing engines in the subset of the plurality of serially connected display data processing steps that are to be performed by the display control device.
Preferably, the display control device is configured to receive the selected shader program from the host device if the display control device does not already have the selected shader program stored in memory.
According to a third aspect, the invention provides a display system comprising a host device as described above, a plurality of display control devices and a plurality of display devices, at least two of the display devices having different characteristics, each display control device of the plurality of display control devices being connected to the host device and to a respective display device, each display control device of the plurality of display control devices being as described above.
According to a fourth aspect, the invention provides a method of displaying display data on a display system comprising a host device, a plurality of display control devices and a plurality of display devices, at least two of the display devices having different characteristics, each display control device of the plurality of display control devices being connected to the host device and to a respective display device, wherein the display data is processed in a display data processing pipeline receiving an initial input of coded display data at the host device and generating a final output of rendered display data at each display control device for transmittal to a respective display device for display, the display data processing pipeline including a plurality of serially connected display data processing steps performed on the display data, the method comprising: determining, at the host device, the characteristics of each of the display devices connected to each of the display control devices to determine characteristics of the rendered display data to be transmitted from the final output of the display data processing pipeline to the particular display device; determining, at the host device, processing capabilities of each of the display control devices connected to the host device; determining, at the host device, based on at least the determined characteristics of the rendered display data required for a particular display device and on the processing capabilities of a particular display control device connected to the particular display device, a subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device, the subset ending at the final output; performing, at the host device, all the display data processing steps from the initial input to a display data processing step performed prior to the subset of the plurality of serially connected display data processing steps to generate part-processed data; and outputting, from the host device, the part-processed display data to the particular display control device to enable the particular display control device to perform the subset of the plurality of serially connected display data processing steps ending at the final output.
Preferably, each display data processing step is performed on a corresponding display data processing engine executing a selected shader program.
In one embodiment, the method further comprises: determining, by the host device, a number of display data processing engines located on the particular display control device based on the processing capabilities of the particular display control device to determine the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device.
Preferably, the method further comprises: selecting, by the host device, a shader program for each of the display data processing engines in the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device; and transmitting, by the host device, instructions to the particular display control device to control the particular display control device to execute the selected shader program on the corresponding display data processing engine to process the display data.
The method may further comprise: transmitting, by the host device, the selected shader program to the particular display control device if the particular display control device does not already have the selected shader program stored in memory.
Preferably, the method further comprises: transmitting, by the display control device to the host device, the characteristics of the display device connected to the display control device to enable the host device to determine characteristics of the rendered display data to be transmitted from the final output of the display data processing pipeline to the particular display device; transmitting, by the display control device to the host device, the processing capabilities of the display control device; receiving, by the display control device from the host device, the information regarding the subset of the plurality of serially connected display data processing steps that are to be performed by the display control device; receiving, by the display control device from the host device, the part-processed display data; performing, by the display control device, the display data processing steps in the subset of the plurality of serially connected display data processing steps on the part-processed display data to generate the rendered display data; and transmitting, by the display control device, the rendered display data to the display device connected to the display control device.
In an embodiment, the method further comprises: receiving, at the display control device, the instructions from the host device to execute the selected shader program on the corresponding display data processing engine to process the display data.
Preferably, the method further comprises: receiving, at the display control device, the selected shader program from the host device if the display control device does not already have the selected shader program stored in memory.
In another aspect, the invention provides a computer readable medium including executable instructions which, when executed in a processing system, cause the processing system to perform all the steps of a method as described above.
According to a further aspect, there is provided a method of co-ordinating a split display pipeline in which incompletely processed display data may be sent from a host to a display control device with a programmable processor and the display control device may then perform the remaining processing to prepare the display data for display. The method comprises: 1. The display control device notifying the host of its processing capabilities 2. The host carrying out some processing and/or data generation 3. The host transmitting appropriate shaders to the display control device 4. The host transmitting data and commands to the display control device 5. The display control device carrying out the instructions in the appropriate shader(s) on the data 6. The display control device transmitting the data to a connected display device for display.
The capabilities of the display control device may include the processing power of the programmable processor, the memory available for shaders, the presence of any specialised hardware, the bandwidth available between the display control device and the display device, or any similar measure of ability to carry out additional processing.
The commands sent to the display control device by the host may include instructions to use a particular shader for particular data, as well as any appropriate flags and parameters, such as which of multiple display devices is or are to be used for displaying the finished image data.
The data sent to the display control device may be display data at any appropriate point in generation and processing. It may include, for example, vertices, primitives, and textures.
As is suggested by the transmission of data and commands to the display control device once the shaders have been transmitted, each shader is uploaded to the display control device once and then used repeatedly. This may be done at the beginning of the link, or repeated throughout the life of the link if, for example, more capability becomes available on the display control device and it therefore becomes possible to upload further shaders in addition to those already present.
An example embodiment could comprise the following steps: 1. The host registering a shader using a unique identifier; 2. The host querying the resources of the display control device to ensure it can run the shader; 3. If so, the host uploading the shader to the display control device and notifying it of the shader’s unique identifier; 4. The host sending primitive or partially-processed data to the display control device for processing with the shader, using the shader’s unique identifier; 5. The display control device processing the data using the shader; 6. The display control device sending the data for display.
The method described above may reduce the amount of processing carried out on the host, as well as providing a potential compression advantage depending on the stage in the pipeline where data is sent to the display control device. Because less processing may be carried out on the host, the host is more likely to be able to supply multiple display control devices with data, especially if they are showing similar images, as is likely in a desktop setting. Furthermore, if the host device were a battery powered device, such as a mobile device, but the display control device was main-powered, or at least with a greater degree of power available than the host device, then off-loading more of the pipeline to the display control device would result in power savings for the host device and therefore in battery life. Indeed, the amount of battery life left for the mobile may even be considered by the host device when determining how much of the pipeline to offload to the display control device.
As an extension of the above method, multiple hosts may be connected to a single display control device. Provided the hosts are aware of one another and are able to reference the appropriate shaders without confusion, the same process can be followed for each connection between a host and the display control device. The display control device need not be aware that the data and accompanying instructions and requests are sent by multiple hosts, as this case is the same as if they were sent by multiple programs on a single host.
Brief Description of the Drawings
Embodiments of the invention will now be more fully described, by reference to the drawings, of which:
Figure 1 shows an example system comprising a host, two display control devices, and connected display devices;
Figure 2 shows a basic schematic of relevant functionality on a host and one display control device arranged according to the current art;
Figure 3 shows a basic schematic of relevant functionality on a host and one display control device in an arrangement by which the pixel shader has been offloaded to the display control device;
Figure 4 shows a basic schematic of relevant functionality on a host and one display control device in an arrangement by which the geometry and pixel shaders have been offloaded to the display control device;
Figure 5 shows a basic schematic of relevant functionality on a host and one display control device in an arrangement by which the vertex, geometry, and pixel shaders have been offloaded to the display control device;
Figure 6 shows a basic schematic of an example host and display control device showing relevant memory and processors; and
Figure 7 shows an overview of an example process with reference to Figure 6.
Detailed Description of the Drawings
Figure 1 shows a host [11] arranged to produce display data for display according to internal instructions. This function will most likely be carried out in the Graphics Processing Unit (GPU) [14] of the host [11], but may be in any internal processor or processors. It is connected to two display control devices [12] which may be embodied, for example, as display adapters. The connections between the host [11] and the display control devices [12] may be wired or wireless, and may utilise a display-specific connection method (such as DVI, or HDMI) or a general-purpose one, such as, for example, a Universal Serial Bus (USB). The latter is more likely, as a computing device is more likely to have output ports for general-purpose data than display data, but the methods of the invention can be applied in either case. The connection may also be over a network, including the internet, and different connections may use different methods.
Each display control device [12] contains its own internal processor [15], as well as a memory [16], in which instructions and data are stored. These are transmitted by the host [11] across the connection and directed to memory [16] for storage. Commands are also transmitted across the connection, but these are sent directly to the processor [15], which then fetches the required data and instructions from the memory [16], Naturally, especially if multiple display data processing steps are being carried out on the display control device [12] then data can be returned to memory [16] in between instructions, but the direction of data flow shown in Figure 1 assumes that data and instructions are being fetched from memory [16] and not returned to it.
The processor [15] is likely to consist of multiple processing cores so that instructions can be run in parallel, and may in fact consist of multiple separate processors. However, for simplicity, the processor [15] is described herein as a single processor [15] regardless of its actual arrangement. As suggested by the fact that instructions may be run in parallel, when additional shaders are present on the display control device [12], different processor cores may simultaneously be running the shaders associated with different display data processing steps, but the processes described herein will refer to a single serially-arranged pipeline.
Furthermore, the processor [15] may consist of or include a collection of display data processing engines, each of which is specifically designed to carry out a particular purpose and run a particular corresponding type of shader. Such engines will be described again in more detail with reference to later Figures, but are collectively described as the processor [15]·
Each display control device [12] is then connected to a display device [13] The display devices [13] are likely to have different characteristics, such as size, resolution, frame update speed, colour depth, etc., meaning that display data destined for the two displays will require different processing. When processing of data is complete, the resulting image data is transmitted along this connection for display on the display device [13], This connection will carry display-specific data, but it may also be wired or wireless. Any one or more of the devices [11, 12, 13] may be co-located with another one or more, where two co-located devices share a single casing and appear to be one device.
Figure 2 shows a host [11] and one of the display control devices [12] arranged according to the current art. For completeness, the respective connected display device [13] is also shown.
Figure 2 also shows steps in an example display data processing pipeline as separate software components or engines. These steps will be used throughout Figures 2, 3, 4, and 5, and may each be represented by a shader. 1. Generation of primitive data, including vertices, according to instructions on the host 2. Vertex shading: performing functions such as rotation and scaling on primitive vertices 3. Geometry shading: Modification of the vertices and primitives prior to the application of textures 4. Texture sampling: Selection and modification of pre-generated textures, including colours 5. Pixel shading: Application of textures to vertices in order to produce pixel data
At the end of the pipeline, the pixel data is rendered and output to the display device.
This pipeline is an example for convenience only. Other pipelines may include more, fewer, or different steps.
Conventionally, the majority of display data processing is carried out on the host [11], as shown in Figure 2, and correspondingly the host may have a complete collection of the required display data processing engines. The various display data processing steps shown here will most likely be run in the GPU [14] on the host [11], either on programmable processors or in specific-function engines as previously mentioned, while data is stored in a memory.
Vertices [21] and primitives [22] are generated and stored in memory. Vertices [21] are collections of points stored as co-ordinates, and these are passed to the vertex shader [25], which transforms the vertices as appropriate. Primitives [22] are parts of display data with dimensionality, such as simple shapes which can then be tessellated in order to produce patterns. This is done by the geometry shader [26], with input from the vertex shader [25], as shown in Figure 2.
The GPU also generates textured surfaces [23], which are transformed textures that are then stored in memory. These are then passed to the texture sampler [24] for sampling, and the resulting textures are combined with the output of the geometry shader [26] in the pixel processing unit [27], This then produces display data, which is transmitted to the display control device [12] and rendered in its internal processor [15] by a pixel rendering service [28], This produces image data, which is sent to the display device [13] for display.
The host [11] shown in all other Figures could have the same capabilities as that shown in Figure 2, including appropriate display data processing engines, and use different capabilities as the situation requires. However, for clarity, only those display data processing steps actually being run on the host [11] in each case will be shown.
Figure 3 shows a similar host [11], display control device [12], and display device [13], but in this case they are arranged according to an embodiment of the invention. The vertex [21] and primitive [22] data are still stored in memory on the host [11], and the vertex [25] and geometry [26] shading are still carried out in the GPU [14] as previously described. However, a subset of the display data processing pipeline has been offloaded to the display control device, together with appropriate partially-processed data.
After being generated on the host [11] the textured surfaces [31] have been sent to the display control device [12] and are stored in memory [16], This means that when the geometry shading [26] is complete, the shaded data can be transmitted to the display control device [12], where pixel shading [27] is carried out, since the textures [31] are also sampled [24] on the display control device [12], The display data can then be rendered into image data and transmitted to the display device [13] as if it had been generated and processed in the conventional way.
This reduces the amount of processing that must be carried out on the host [11] and may result in a smaller volume of data being transmitted to the display control device [12] if, for example, textures have not yet been applied at the time the data is transmitted. This may be beneficial where the bandwidth of the connection between the host [11] and the display control device [12] is limited. Compression may be applied to the data as it is transmitted from the host [11] to the display control device [12], which is not done at this stage in the pipeline in the current art as conventionally the whole pipeline is contained on a single device. In many cases, however, there will be greater benefits from compression partway through the pipeline, meaning that a system arranged according to the invention may make more efficient use of a limited-bandwidth connection.
Figure 4 shows a further embodiment of a system according to the invention. In this case, both the textured surfaces [31] and the primitives [41] have been transmitted to the display control device [12], and these are stored in memory [16] on the display control device [12], The instructions comprising the geometry shader [26] have also been transmitted. As a result, the geometry shader [26], texture sampler [24], and pixel shader [27] are all run by the processor [15] on the display control device [12], prior to the resulting display data being rendered and sent as image data to the display device [13] in the conventional way. Only the vertex shader [25] is run in the GPU [14] on the host [11],
Figure 5 shows a final embodiment, in which all the shaders [24-7] are run on the processor [15] in the display control device [12], As a result, both the vertices [51] and the primitives [41], having been generated [21, 22] on the host [11], are transmitted to the display control device [12] and stored in memory [16] along with the textured surfaces [31], The instructions comprising the vertex, geometry, and pixel shaders [25, 26, 27] and the texture sampler [24] are also transmitted and these are run on the data [31, 41, 51] stored in memory [16] as previously described with regard to the host [11], Finally, the display data is rendered to image data and transmitted to the display device [13] in the conventional way.
In each of Figures 3, 4, and 5, only one display control device [12] is shown. Different levels of offloading may be used by the host [11] for different display control devices [12] in the same system, such that in the system shown in Figure 1 where there are two display control devices [12], The interaction between the host [11] and the first display control device [12] may be like that shown in Figure 3, in which only the pixel shading step [27] is offloaded to the display control device [12], Meanwhile, the interaction between the host [11] and the second display control device [12] may be like that shown in Figure 5, in which all the display data processing is performed on the display control device [12] apart from the initial generation of the primitives [21, 22], Naturally, there may also be other combinations, including both display control devices [12] performing the same level of processing, perhaps using different shaders, and the pipeline may be split in other ways as appropriate.
Figure 6 shows an example of a basic schematic of a host [61], display control device [62], and display device [63], This schematic may run the relevant display data processing steps in any configuration shown in any of Figures 2, 3, 4, or 5.
The host [61] will include other components, but those relevant to the invention are a memory [64] which contains shaders and a processor [65], which may, as previously mentioned, actually comprise a collection of one or more display data processing engines: possibly all those required for the full pipeline as shown in Figure 2. In this example, the host processor [65] handles the connection and all communication with the display control device [62], although in practice there is likely to also be an output engine that handles the actual communication by, for example, converting data to an appropriate communication format and transmitting it. For the purposes of this description, data is sent directly from the host processor [65] to the display control device [62],
The display control device [62] is also likely to include other components, but only three are shown in Figure 6 for the purposes of clarity. These are, first, an input engine [66], which receives signals and data from the host [61] and handles them as appropriate, for example by sending further signals or removing any transmission formatting added by the host [61], such as packet headers. The input engine [66] is then connected to a memory [67] and a processor [68], The memory [67] holds shaders [610], along with their unique reference numbers as assigned by the host [61], and data [69], The shaders [610] may in fact be represented by a lookup table referencing a further, larger memory where the full instructions comprising the shaders are stored while the local memory [67] only holds the unique references and the memory addresses of the instructions, but in the embodiment shown in Figure 6 the instructions comprising the shaders [610] are stored in the local memory [67], The memory [67] is also connected to the processor [68] so that the processor [68] can fetch data [69] and instructions [610] from the memory [67] and may also store data [69] in the memory [67] if more processing is required. These two connections are shown as separate for clarity, but in practice are likely to be a single bus.
The display control device processor [68] runs the instructions comprised in shaders on data received by the input engine [66] and stored in memory [69] or passed directly to the processor [68], The processor may, as previously mentioned, actually comprise a collection of one or more display data processing engines. Unlike the host [61], the display control device [62] does not produce primitives and is highly unlikely to have a full collection of display data processing engines, so it is likely to only be able to perform some of the display data processing steps.
This then produces processed data, which the processor [68] either transmits to the display device [63] for display in the conventional way, or stores in memory [69] so that it can fetch it and run further instructions on it as appropriate. Transmitting finished image data to the display device [63] may involve an output engine to apply formatting and timing appropriate to the display device [63],
Figure 7 shows an overview of an example of the whole process for the host to offload the operation of a shader.
At Step S71, the host [61] generates the primitive image data [22] it will require. As previously mentioned, this may consist of lines, simple shapes such as triangles, and colours. These are stored locally on the host [61],
The host [61] also has at least one shader [64], consisting of instructions to be carried out on display data during a particular display data processing step. It registers the shaders at Step S72, assigning each one a reference number. Examples are shown in Figure 6: Shader A [64A] has been assigned reference number 1, Shader B [64B] has reference number 2, Shader C [64C] has reference number 3, and Shader D [64D] has reference number 4. If there were multiple hosts [61] connected to a single display control device [62], these identification numbers would also be unique between hosts [61] unless it could be guaranteed that all hosts [61] would use an identical shader.
At Step S73, the host [61] queries the display control device [62] to see what resources it has available for running shaders, as well as to determine the characteristics of the connected display device or devices [63] in order to determine the characteristics of the data that will be sent to them. The available resources on the display control device [62] would include memory for the instructions [610] and the processing power required for an additional program, and also memory [69] available for the inputs required by the program. If appropriate, it could also include availability of display data processing engines. For example, even in the embodiment shown in Figure 5 where the display control device [12] would be capable of running all the shaders required by the host [11], it will not be possible to offload all the processing unless there is also memory space available for the vertices [51] and primitives [41], together with all the intermediate stages of the data. As a result, all of these attributes must be checked during querying.
Alternatively, the display control device [62] could transmit its capabilities to the host [61] without querying, for example upon connection. This would mean that the appropriate data would already be stored on the host [61] and it could consult this at Step S73.
In the example shown in Figure 6, the host [61] is querying for the possibility of offloading Shader D [64D], There is sufficient space in memory [610] on the display control device [62] for an additional shader to be stored, and there is also sufficient processing power and other memory available for it to be run. The host [61] is therefore able to upload the instructions comprising Shader D [64D] to the display control device [62] at Step S74.
The processor [65] on the host [61] fetches the shader [64D] from memory [64] and transmits it to the input engine [66] on the display control device [62], which immediately directs it into the memory [67] on the display control device [62], where it is stored in the area of memory dedicated to program instructions [610], The host [61] also sends the unique identifier associated with Shader D [64D] - ‘4’, in this example - and this is stored in memory [610] with the shader. It may also be stored with a reference to the shader such as a memory address in a look-up table to provide easy access to each shader using its unique identifier.
Decisions regarding the display data processing steps to be offloaded could be dictated by the characteristics of all the connected display devices [63] such that if they all have particular attributes in common then relevant steps could be most efficiently carried out centrally on the host [61] even if all the display control devices [62] would in fact be capable of performing them. Furthermore, if there are three connected display devices [63] and one is different to the others - for example, it is much larger - the host [61] may determine that a larger subset of the display data processing steps should be carried out on that display device’s [63] display control device [62], so that scaling common to the remaining two display devices [63] is performed on the host [61] and the display control device [62] connected to the large display device [63] performs its scaling. In this way, the system is highly flexible for greatest efficiency.
Furthermore, by splitting the pipeline in different ways, different compression methods may be used according to which compression method may be particularly appropriate for the particular partially-processed display data being transferred across from the host to the display control device. Indeed, in some circumstances the host may determine that, although the pipeline could be split in one way so that some processing is offloaded to the display control device as described above, by splitting the pipeline in a different place, better compression may be obtained depending on the partially processed data that is then to be sent, and that such better compression may be preferable to the offloading that would otherwise have been made. Thus, the host could use knowledge of the various compression methods available for different partially processed data and capabilities of the transmission pipeline, such as bandwidth and available resources, when determining where to split the pipeline.
The example being outlined in Figure 7 assumes a system such as that shown in Figure 5, in which all shading is offloaded to the display control device [12], As a result, at Step S75, the host [61/11] sends the generated primitives [22] to the display control device [62/12], where they are stored in memory [51], In another system, it will transmit partially-shaded display data. The incoming data will be stored in memory [69] on the display control device [62/12] at Step S76.
In any case, the host [61] will perform any display data processing steps it has determined should not be offloaded to the display control device [62], and will transmit the part-processed display data to the display control device [62] as described above for the primitive data.
At Step S77, the host [61] transmits the identifiers of the shaders to be applied to the data, preferably in the order in which they are to be applied, but a different ordering scheme may be used. For example, if Shaders A and B are vertex shaders with slightly different functions but which occupy the same display data processing step, Shader C is a geometry shader, and Shader D is a pixel shader, the host [61] may only transmit ‘4’ for a pipeline such as that shown in Figure 3, in which only pixel processing [27] is carried out on the display control device [12], Alternatively, it may send the data accompanied by ‘1, 3, 4’ for a pipeline such as that shown in Figure 5 using Shader A for vertex shading [25], Shader C for geometry shading [26], and Shader D for pixel shading [27], This introduces flexibility in which shaders are applied in different circumstances.
At Step S78, the processor [68] on the display control device [62] fetches the referenced shader [610] from memory [67] and carries out the instructions contained in it on the primitives or incomplete data transmitted to it, having also fetched these from memory [69], If there are multiple shaders to be applied, it will repeat this step for all the shaders in the sequence and may move the data in and out of memory [69] between iterations. It may also send feedback to the host [61] on its progress in processing the data, which may allow the host [61] to balance workload, for example between itself and the display control device [62], This step may involve sending the shaders and data to appropriate display data processing engines within the processor [68], or carrying out the instructions in each shader using a programmable processor [68],
At Step S79, the processing is complete and the display control device [62] transmits the data to the display device [63] for display in the conventional way.
Although particular embodiments have been described in detail above, it will be appreciated that various changes, modifications and improvements can be made by a person skilled in the art without departing from the scope of the present invention as defined in the claims. For example, hardware aspects may be implemented as software where appropriate and vice versa, and engines/modules which are described as separate may be combined into single engines/modules and vice versa. Functionality of the engines or other modules may be embodied in one or more hardware processing device(s) e.g. processors and/or in one or more software modules, or in any appropriate combination of hardware devices and software modules. Furthermore, software instructions to implement the described methods may be provided on a computer readable medium.
Claims (29)
1. A host device for use in a display system for displaying display data, the display system comprising a host device, a plurality of display control devices and a plurality of display devices, at least two of the display devices having different characteristics, each display control device of the plurality of display control devices being connected to the host device and to a respective display device, wherein the display data is processed in a display data processing pipeline receiving an initial input of coded display data at the host device and generating a final output of rendered display data at each display control device for transmittal to a respective display device for display, the display data processing pipeline including a plurality of serially connected display data processing steps performed on the display data, wherein the host device is configured to: determine the characteristics of each of the display devices connected to each of the display control devices to determine characteristics of the rendered display data to be transmitted from the final output of the display data processing pipeline to the particular display device; determine processing capabilities of each of the display control devices connected to the host device; determine, based on at least the determined characteristics of the rendered display data required for a particular display device and on the processing capabilities of a particular display control device connected to the particular display device, a subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device, the subset ending at the final output; perform all the display data processing steps from the initial input to a display data processing step performed prior to the subset of the plurality of serially connected display data processing steps to generate part-processed data; and output the part-processed display data to the particular display control device to enable the particular display control device to perform the subset of the plurality of serially connected display data processing steps ending at the final output.
2. A host device according to claim 1, wherein each display data processing step is performed on a corresponding display data processing engine executing a selected shader program.
3. A host device according to claim 2, wherein the host device comprises all the display data processing engines required for the display data processing pipeline.
4. A host device according to either claim 2 or claim 3, wherein the host device is configured to determine a number of display data processing engines located on the particular display control device based on the processing capabilities of the particular display control device to determine the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device.
5. A host device according to claim 4, wherein the host device is configured to select a shader program for each of the display data processing engines in the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device and to transmit instructions to the particular display control device to control the particular display control device to execute the selected shader program on the corresponding display data processing engine to process the display data.
6. A host device according to claim 5, wherein the host device is configured to transmit the selected shader program to the particular display control device if the particular display control device does not already have the selected shader program stored in memory.
7. A host device according to any preceding claim, wherein the part-processed display data is output to the particular display control device in compressed form.
8. A host device according to claim 7, wherein the host device is further configured to: determine available compression methods for part-processed data at different display data processing steps in the host device and the amount of compressed part-processed display data that would be output if part-processed data at different display data processing steps in the host device were compressed according to different available compression methods; utilise the knowledge of the amount of compressed part-processed display data that would be output from different display data processing steps in the host device compressed according to different available compression methods when determining the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device.
9. A display control device for use in a display system, the display system comprising a host device, a plurality of display control devices and a plurality of display devices, at least two of the display devices having different characteristics, each display control device of the plurality of display control devices being connected to the host device and to a respective display device, wherein the display data is processed in a display data processing pipeline receiving an initial input of coded display data at the host device and generating a final output of rendered display data at each display control device for transmittal to a respective display device for display, the display data processing pipeline including a plurality of serially connected display data processing steps performed on the display data, wherein the display control device is configured to: transmit to the host device the characteristics of the display device connected to the display control device to enable the host device to determine characteristics of the rendered display data to be transmitted from the final output of the display data processing pipeline to the particular display device; transmit to the host device processing capabilities of the display control device; receive, from the host device, information regarding a subset of the plurality of serially connected display data processing steps that are to be performed by the display control device, the subset ending at the final output; receive, from the host device, part-processed display data to enable the display control device to perform the subset of the plurality of serially connected display data processing steps; perform the display data processing steps in the subset of the plurality of serially connected display data processing steps on the part-processed display data to generate the rendered display data; and transmit the rendered display data to the display device connected to the display control device.
10. A display control device according to claim 9, wherein each display data processing step is performed on a corresponding display data processing engine executing a selected shader program.
11. A display control device according to claim 10, wherein the display control device comprises fewer than all the display data processing engines required for the display data processing pipeline.
12. A display control device according to either claim 10 or claim 11, wherein the display control device is configured to receive instructions from the host device to execute a selected shader program on a corresponding display data processing engine to process the display data, the selected shader program being selected by the host device for each of the display data processing engines in the subset of the plurality of serially connected display data processing steps that are to be performed by the display control device.
13. A display control device according to claim 12, wherein the display control device is configured to receive the selected shader program from the host device if the display control device does not already have the selected shader program stored in memory.
14. A display system comprising a host device according to any one of claims 1 to 8, a plurality of display control devices and a plurality of display devices, at least two of the display devices having different characteristics, each display control device of the plurality of display control devices being connected to the host device and to a respective display device, each display control device of the plurality of display control devices being according to any one of claims 9 to 13.
15. A method of displaying display data on a display system comprising a host device, a plurality of display control devices and a plurality of display devices, at least two of the display devices having different characteristics, each display control device of the plurality of display control devices being connected to the host device and to a respective display device, wherein the display data is processed in a display data processing pipeline receiving an initial input of coded display data at the host device and generating a final output of rendered display data at each display control device for transmittal to a respective display device for display, the display data processing pipeline including a plurality of serially connected display data processing steps performed on the display data, the method comprising: determining, at the host device, the characteristics of each of the display devices connected to each of the display control devices to determine characteristics of the rendered display data to be transmitted from the final output of the display data processing pipeline to the particular display device; determining, at the host device, processing capabilities of each of the display control devices connected to the host device; determining, at the host device, based on at least the determined characteristics of the rendered display data required for a particular display device and on the processing capabilities of a particular display control device connected to the particular display device, a subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device, the subset ending at the final output; performing, at the host device, all the display data processing steps from the initial input to a display data processing step performed prior to the subset of the plurality of serially connected display data processing steps to generate part-processed data; and outputting, from the host device, the part-processed display data to the particular display control device to enable the particular display control device to perform the subset of the plurality of serially connected display data processing steps ending at the final output.
16. A method according to claim 15, wherein each display data processing step is performed on a corresponding display data processing engine executing a selected shader program.
17. A method according to claim 16, further comprising: determining, by the host device, a number of display data processing engines located on the particular display control device based on the processing capabilities of the particular display control device to determine the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device.
18. A method according to claim 17, further comprising: selecting, by the host device, a shader program for each of the display data processing engines in the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device; and transmitting, by the host device, instructions to the particular display control device to control the particular display control device to execute the selected shader program on the corresponding display data processing engine to process the display data.
19. A method according to claim 18, further comprising: transmitting, by the host device, the selected shader program to the particular display control device if the particular display control device does not already have the selected shader program stored in memory.
20. A method according to any one of claims 15 to 19, further comprising: transmitting, by the display control device to the host device, the characteristics of the display device connected to the display control device to enable the host device to determine characteristics of the rendered display data to be transmitted from the final output of the display data processing pipeline to the particular display device; transmitting, by the display control device to the host device, the processing capabilities of the display control device; receiving, by the display control device from the host device, the information regarding the subset of the plurality of serially connected display data processing steps that are to be performed by the display control device; receiving, by the display control device from the host device, the part-processed display data; performing, by the display control device, the display data processing steps in the subset of the plurality of serially connected display data processing steps on the part-processed display data to generate the rendered display data; and transmitting, by the display control device, the rendered display data to the display device connected to the display control device.
21. A method according to claim 20, further comprising: receiving, at the display control device, the instructions from the host device to execute the selected shader program on the corresponding display data processing engine to process the display data.
22. A method according to claim 21, further comprising: receiving, at the display control device, the selected shader program from the host device if the display control device does not already have the selected shader program stored in memory.
23. A method according to one of claims 15 to 22, further comprising compressing the part-processed display data prior to outputting it to the particular display control device in compressed form.
24. A method according to claim 23, wherein the method further comprises: determining available compression methods for part-processed data at different display data processing steps in the host device and the amount of compressed part-processed display data that would be output if part-processed data at different display data processing steps in the host device were compressed according to different available compression methods; utilising the knowledge of the amount of compressed part-processed display data that would be output from different display data processing steps in the host device compressed according to different available compression methods when determining the subset of the plurality of serially connected display data processing steps that are to be performed by the particular display control device.
25. A computer readable medium including executable instructions which, when executed in a processing system, cause the processing system to perform all the steps of a method according to any one of claims 15 to 24.
26. A host device substantially as hereinbefore described with reference to the drawings.
27. A display control device substantially as hereinbefore described with reference to the drawings.
28. A display system substantially as hereinbefore described with reference to the drawings.
29. A method of displaying display data on a display system substantially as hereinbefore described with reference to the drawings.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| GB1609605.9A GB2550942B (en) | 2016-06-01 | 2016-06-01 | Display Data processing |
| PCT/GB2017/051509 WO2017207971A1 (en) | 2016-06-01 | 2017-05-26 | Display data processing |
| US16/306,520 US10481852B2 (en) | 2016-06-01 | 2017-05-26 | Display data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| GB1609605.9A GB2550942B (en) | 2016-06-01 | 2016-06-01 | Display Data processing |
Publications (3)
| Publication Number | Publication Date |
|---|---|
| GB201609605D0 GB201609605D0 (en) | 2016-07-13 |
| GB2550942A true GB2550942A (en) | 2017-12-06 |
| GB2550942B GB2550942B (en) | 2021-11-10 |
Family
ID=56410854
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| GB1609605.9A Active GB2550942B (en) | 2016-06-01 | 2016-06-01 | Display Data processing |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US10481852B2 (en) |
| GB (1) | GB2550942B (en) |
| WO (1) | WO2017207971A1 (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101771242B1 (en) * | 2014-08-29 | 2017-08-24 | 서강대학교산학협력단 | High-speed parallel processing method of ultrasonics wave signal using smart device |
| EP3472806A4 (en) | 2016-06-17 | 2020-02-26 | Immersive Robotics Pty Ltd | Image compression method and apparatus |
| AU2018217434C1 (en) | 2017-02-08 | 2023-04-27 | Immersive Robotics Pty Ltd | Displaying content to users in a multiplayer venue |
| AU2018372561B2 (en) | 2017-11-21 | 2023-01-05 | Immersive Robotics Pty Ltd | Image compression for digital reality |
| AU2018373495B2 (en) | 2017-11-21 | 2023-01-05 | Immersive Robotics Pty Ltd | Frequency component selection for image compression |
| TWI708976B (en) * | 2019-08-13 | 2020-11-01 | 華碩電腦股份有限公司 | Display device and control method thereof |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2530582A1 (en) * | 2011-06-04 | 2012-12-05 | Apple Inc. | Adaptive use of wireless display |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040075623A1 (en) | 2002-10-17 | 2004-04-22 | Microsoft Corporation | Method and system for displaying images on multiple monitors |
| JP4790227B2 (en) | 2004-04-05 | 2011-10-12 | パナソニック株式会社 | Display control apparatus and display control method |
| US20090160731A1 (en) * | 2007-12-20 | 2009-06-25 | Motorola, Inc. | Method for clustering displays of display devices |
| US9424586B1 (en) * | 2013-09-18 | 2016-08-23 | American Megatrends, Inc. | Remote sensor management |
| US10373282B2 (en) * | 2013-11-18 | 2019-08-06 | Samsung Electronics Co., Ltd. | Method and system for controlling display parameters through multiple pipelines |
| GB201410314D0 (en) | 2014-06-10 | 2014-07-23 | Advanced Risc Mach Ltd | Display controller |
-
2016
- 2016-06-01 GB GB1609605.9A patent/GB2550942B/en active Active
-
2017
- 2017-05-26 US US16/306,520 patent/US10481852B2/en active Active
- 2017-05-26 WO PCT/GB2017/051509 patent/WO2017207971A1/en active Application Filing
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2530582A1 (en) * | 2011-06-04 | 2012-12-05 | Apple Inc. | Adaptive use of wireless display |
Also Published As
| Publication number | Publication date |
|---|---|
| US20190179589A1 (en) | 2019-06-13 |
| US10481852B2 (en) | 2019-11-19 |
| GB201609605D0 (en) | 2016-07-13 |
| GB2550942B (en) | 2021-11-10 |
| WO2017207971A1 (en) | 2017-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10481852B2 (en) | Display data processing | |
| CN111033570B (en) | Rendering images from computer graphics using two rendering computing devices | |
| US10152765B2 (en) | Texture processing method and unit | |
| US7173635B2 (en) | Remote graphical user interface support using a graphics processing unit | |
| US20100134494A1 (en) | Remote shading-based 3d streaming apparatus and method | |
| US7187384B2 (en) | Systems and methods for downloading custom or prefabricated processing routines to programmable graphics chips | |
| US10388033B2 (en) | Texture processing apparatus and method | |
| US6628277B1 (en) | Decompression of three-dimensional graphics data using mesh buffer references to reduce redundancy of processing | |
| CN105096373A (en) | Media content rendering method, user device and rendering system | |
| US20100046846A1 (en) | Image compression and/or decompression | |
| EP3147865A1 (en) | Method and apparatus for processing compressed texture | |
| US10453274B2 (en) | Texture compressing method and apparatus and texture decompressing method and apparatus | |
| JP2016062594A (en) | Method and apparatus for processing texture | |
| CN115794913B (en) | Data processing method and device in artificial intelligence system | |
| US20240096007A1 (en) | Rendering Method, Device, and System | |
| US7336284B2 (en) | Two level cache memory architecture | |
| EP3156974A1 (en) | Texture processing method and unit | |
| KR100806345B1 (en) | 3-dimensional graphics accelerator and method reading texture data | |
| US20150054843A1 (en) | Color-correct alpha blending texture filter and method of use thereof | |
| CN109087381B (en) | Unified architecture rendering shader based on dual-emission VLIW | |
| US10062140B2 (en) | Graphics processing systems | |
| CN111462289B (en) | Image rendering method, device and system | |
| Colantoni et al. | Web technologies enable agile color management | |
| US11423588B2 (en) | Color transforms using static shaders compiled at initialization | |
| Ikkala et al. | DDISH-GI: Dynamic Distributed Spherical Harmonics Global Illumination |