KR102561042B1 - Data processing systems - Google Patents

Abstract

The display controller 93 in the data processing system operates to perform time warp processing of the rendered frame 92 generated by the graphics processor (GPU) 91 for presentation to the display panel 94. A possible time warp module (transformation stage) 100 is included. Time warp module (transformation stage) 100 uses the received view orientation to provide a suitable "time warped" transformed version of the input surface as an output transformed surface for display on display 94. It operates to transform the input surface 92 read by the display controller 93 based on the data.

Description

Data processing system {DATA PROCESSING SYSTEMS}

The present invention relates to data processing systems and, more particularly, to the operation of data processing systems for displaying images on a display.

1 shows a host processor including a central processing unit (CPU) 7, a graphics processing unit (GPU) 2, a video codec 1, a display controller 5 and An exemplary data processing system including a memory controller 8 is shown. As shown in FIG. 1 , these units communicate via an interconnection 9 and access an off-chip memory 3 . In this system, the GPU 2, the video codec 1 and/or the CPU 7 create frames (images) to be displayed, and then the display controller 5 sends the frames to the display 4 for display. to provide.

In use of this system, an application 10 such as a game running on the host processor (CPU) 7 will require display of frames on the display 4, for example. To do this, the application 10 will submit appropriate commands and data to the driver 11 for the graphics processing unit 2 running on the CPU 7 . The driver 11 then uses suitable commands and commands to cause the graphics processing unit 2 to render suitable frames for display and to store these frames in suitable frame buffers, for example in main memory 3 . will generate data. The display controller 5 will then read these frames into a buffer for display, which is then read out therefrom and displayed on the display panel of the display 4.

9 illustrates this, showing GPU 91 generating rendered frames 92 which it stores in memory in response to commands and data received from host 90 . As shown in Fig. 9, the rendered frame 92 is then subsequently read by the display controller 5 and provided to the display panel 94 for display.

One example of the use of a data processing system as illustrated in FIG. 1 is to provide a virtual reality (VR) head mounted display (HMD) system. (In this case, display 4 would be a head mounted display of the same type.)

In head-mounted virtual reality display operation, appropriate images to be displayed in each eye respond to appropriate commands and data from applications such as games (e.g., running on CPU 7) that require the virtual reality display. and will be rendered by the graphics processing unit 2. The GPU 2 will render the image for display at a rate that matches the refresh rate of the display, such as 30 frames per second, for example.

In this way, the system will also work to track the movement of the user's head/gaze (so-called head pose tracking). This head orientation (pose) data is then used to determine how the image should actually be displayed to the user for the current head position (view direction), relative to the user's current direction of view. The image (frame) is rendered accordingly (eg, by setting the camera (viewpoint) orientation based on the head orientation data) so that an appropriate image based thereon can be displayed.

In the VR system, the head orientation (pose) can be determined simply at the beginning of rendering the frame to be displayed, but because of the latency in the rendering process, the detection of the head orientation (pose) at the start of rendering of the frame and the frame actually The orientation (pose) of the user's head may be changed between times of display (scanning on the display panel).

To allow for this, a process known as “time warp” has been proposed for virtual reality head mounted display systems. In this process, a frame to be displayed is rendered based on head orientation data detected at the start of rendering of the frame, but before the frame is actually displayed, additional head orientation (pose) data is detected, and then the head pose is updated. The sensor data is used to render an "updated" version of the original frame that takes into account the updated head orientation (pose) data. An "updated" version of the frame is then displayed. This allows the image displayed on the display to more closely match the user's most recent head orientation.

To do this, an initial "application" frame is rendered into an appropriate buffer in memory, but then a version of the initially rendered frame that takes into account the most recent head orientation is rendered to present the frame to be displayed to the user. To do this, there is a second rendering process that takes the initial application frame from memory and uses the most recent head orientation (pose). This typically involves performing some form of transformation on the initial frame based on head orientation (pose) data. This "time warp" rendered frame that is actually displayed is then written to an additional buffer or buffers in memory that is read out for display by the display controller.

Both of these rendering operations are normally performed by the graphics processing unit 2, under suitable control from the CPU 7. Thus, for this process, graphics processing unit 2 will need to perform two different rendering tasks, one for rendering the "application" frame as requested and commanded by the application. and another to then "time warp" render these rendered frames as appropriate based on the most recent head orientation data into a buffer in memory for reading by the display controller 5 for display.

2 and 3 illustrate the “time warp” process.

Figure 2 shows the display of an exemplary frame 20 when the viewer is straight ahead, and the necessary "time warp" projection of that frame 21 when the user's viewing angle changes. From this figure it can be seen that for that frame 21 a modified version of frame 20 should be displayed.

3 shows a time warped rendering 31 of an application frame 30 to correspondingly provide a “time warped” frame 32 for display. As shown in FIG. 3, a given application frame 30 that has been rendered is sent a suitable "time-warped" version 32 of that application frame 30 at successive intervals while waiting for a new application frame to be rendered. It can receive two time warp processes 31 for the purpose of displaying . Figure 3 also shows a typical sampling 33 of head position (pose) data used to determine the appropriate "time warp" modifications that should be applied to the application frame 30 in order to properly display the frame to the user based on the head position. ) is shown.

Applicants believe that there is room for improved ways to perform “time warp” rendering for virtual reality displays in data processing systems.

According to a first aspect of the present invention there is provided a display controller for a data processing system comprising: a local input buffer operable to store data of an input surface;

an input stage operable to fetch data of an input surface from memory into the local input buffer;

an output stage operable to provide an output surface for display on a display; and

identify data regions of the input surface for which data is needed to provide an output transformed version of the input surface, cause the input stage to fetch the identified data regions of the input surface from the memory into the local input buffer, and processing input surface data from the local input buffer to transform data of the input surface based on received view orientation data to provide the output transformed version of the input surface as an output transformed surface; , a conversion stage operable to provide the output transformed surface to the output stage for presentation as an output surface for display to a display.

According to a second aspect of the present invention there is provided a method of operating a display controller in a display processing system comprising: a local input buffer operable to store data of an input surface; An input stage operable to fetch from memory into the local input buffer, transform an input surface based on received view orientation data and provide a transformed version of the input surface as an output transformed surface. Include possible transformation stages,

Way,

delete

The conversion stage of the display controller identifies data regions of the input surface for which data is needed to provide an output transformed version of the input surface, and transfers the identified data regions of the input surface from memory to the local input. Input surface data from the local input buffer to fetch into a buffer and transform the data of the input surface based on received viewing orientation data to provide the output transformed version of the input surface as the output transformed surface. processing; and

providing, by the display controller, the output converted surface to a display for display.

The present invention relates to a display controller operable to provide a display with an output surface for display. As in conventional display controllers, the display controller of the present invention has an input stage operable to read at least one input surface (e.g., a frame), and an input stage operable to provide an output surface (a frame) for display. Include an output stage. However, in contrast to conventional display controllers, the display controller of the present invention transforms the input surface based on received viewing orientation data (e.g., and preferably, head position (pose) (tracking) information). and a transformation stage operable to provide a transformed version of the input surface as an output surface for display.

As discussed further below, the transform stage is an alternative to those operations that need to be performed by the display controller of the present invention, e.g., by a graphics processing unit to provide a “time warped” frame for display. , which means that the input surface itself can perform the transformation process required for temporal warp rendering.

In particular, Applicant believes that display processing operations are performed to provide a frame for display, instead of having to first perform processing elsewhere in the data processing system (e.g., in a graphics processing unit) before processing the frame for display. It has been recognized that it is possible to perform a temporal warp transformation of the frame to be displayed in the display controller as part of the

Performing the time warp conversion process at the display controller while the time warped frame is being displayed provides a number of advantages. For example, this prevents the "time warp" processing of application frames from having to be performed on the GPU and written to memory to be read by the display controller. This can provide significant savings in terms of memory bandwidth, processing resources, and power.

It also eliminates the need for the graphics processing unit to perform temporal warp rendering of application frames for display, whereby, for example, the graphics processing unit performs two rendering processes (one renders the application frame and the other renders the application frame). One avoids the need to do a temporal warp rendering of these application frames) and avoids the need to switch between these two rendering processes. Additionally, the graphics processor can be used exclusively for rendering application frames, thereby eliminating the need to divert graphics processor resources to a “time warp” operation.

In addition, the present invention allows the transformation for time warp processing to be more closely synchronized with the display of frames to the user, so that the display to the user can be very closely aligned with the user's current head position.

Further, the present invention allows temporal warp rendering of application frames displayed in a virtual reality headset to be performed, in substantially real time, while the display stream is being provided to the headset for display, thereby, for example: Instead of having to wait for the next frame to be displayed before any changes can be made, adjustments to the displayed frame based on the user's head movement are applied at a finer resolution, eg on a line-by-line basis. can do.

All of this makes it possible to provide an improved display to the user compared to conventional approaches.

The input surface to which the transformation stage transforms may be any suitable and desired such surface. Preferably, the input surface is an input surface intended to be displayed on a display to which the display controller is associated. In one preferred embodiment, the input surface is an image for display, eg a frame.

In a preferred embodiment, the input surface that the transform stage transforms is created for display for an application such as a game, but initially rendered and then displayed based on the determined viewing orientation (e.g., and preferably, contains frames that must undergo "time warp" processing.

The input surface (and each input surface) will contain an array of data elements (sampling locations) (eg pixels), each of which stores suitable data (eg a set of color values).

The input surface or surfaces can be created as desired.

The input surface(s) are preferably created (rendered) by a graphics processing unit (graphics processor) of the data processing system of which the display controller is a part, but which, in addition or instead, is an integral part, such as a CPU or video processor, if desired. It may be created or provided by another component or components of the data processing system.

The generated input surface is preferably stored, as discussed above, in a memory, for example a frame buffer, which is then read by an input stage of the display controller for processing by the display controller.

The memory in which the input surface is stored may include any suitable memory and may be configured in any suitable and desired manner. For example, it may be an on-chip memory with the display controller or an external memory. In a preferred embodiment, this is an external memory, such as the main memory of the entire data processing system. It can be a memory dedicated for this purpose or it can be a part of memory that can also be used for other data. In one preferred embodiment, at least one input surface is stored in (and read from) a frame buffer (eg, an “eye” buffer).

The input stage of the display controller may include any suitable such stage operable to read data of the input surface from a memory where the data is stored. In one preferred embodiment, the input stage preferably includes a memory read sub comprising a Direct Memory Access (DMA) read controller configured (operable) to read data of the input surface from a memory in which the input surface is stored. contains the system

In one particularly preferred embodiment, the input stage is operable to fetch data of the input surface to be processed by the display controller from memory to a local buffer or buffers of the display controller (thus preferably the display controller is and one or more local buffers operable to store data of the input surface or surfaces to be processed by the controller). The conversion stage then preferably processes the input surface from the local buffer or buffers.

Thus, in one particularly preferred embodiment, the conversion stage has an associated input buffer or buffers into which the data of the input stage to be processed is loaded for processing by the conversion stage.

Correspondingly, in one particularly preferred embodiment, the present invention provides an input stage of the display controller which loads the data of the input surface to be processed from memory into the display controller's local (input) buffer or buffers, and then the received viewing orientation. and a transform stage that processes the local (input) buffer or input surface data from the buffer based on the data to provide an output transformed surface.

The display controller's local (input) buffer or buffers into which input surface data is fetched may be any suitable and desired local buffer memory of the display controller or any suitable and desired local buffer memory accessible to the display controller. As discussed further below, the local buffer or buffers are preferably capable of storing (and operable to store) one or more, preferably a plurality of blocks of data elements (sampling (data) locations) of the input surface. .

A local buffer or buffers may be provided specifically for the purposes of the present invention, or may already exist in a display controller or display controller for another purpose, such as a buffer or buffers in each layer pipeline or pipelines of a display controller. may be a buffer or buffers that already exist for

Thus, in one preferred embodiment, the display controller comprises one or more, preferably multiple, input layer (surface) processing pipelines, one or more of these input layer pipelines defining the input surface to be transformed by the transform stage. The conversion stage is operable to read the required input surface data from a suitable input layer pipeline or buffer or buffers of pipelines.

Also preferably, the overall operation includes first rendering the input surface (eg, by a graphics processing unit) and then storing the input surface in memory to be read by the input stage.

The transform stage is operative to transform the input surface based on the received viewing orientation data.

The field of view orientation data may be any suitable and desired data representing a field of view orientation (field direction). In a preferred embodiment, the viewing orientation data expresses and indicates a desired viewing orientation (viewing direction) from which the input surface is displayed as if viewed from (the input surface to be transformed is displayed with respect to).

In one particularly preferred embodiment, the viewing orientation data is relative to the input surface to a reference (eg, predefined) viewing location (which may but need not be a “straight ahead” viewing location). Indicates the orientation of the viewing position to be displayed as

The reference viewing position is preferably the generated (rendered) viewing position (direction (orientation)) with respect to the input surface.

Thus, in one preferred embodiment, the viewing orientation data indicates the orientation of the viewing position at which the input surface will be displayed relative to the generated (rendered) viewing position (orientation) to which the input surface is associated.

The viewing orientation data preferably represents a rotation of the viewing position at which the input surface is displayed relative to a reference viewing position. View position rotation can be provided as desired, such as in the form of three (Euler) angles or as a quaternion.

Thus, in this particularly preferred embodiment, the viewing orientation data is one or more (and preferably three) angles representing the orientation of the viewing location at which the input surface will be displayed relative to a reference (eg predefined) viewing location. (Euler angles).

View orientation data is provided to the display controller in use in any suitable and desired manner. Preferably, this is suitably provided by applications requiring the display of output converted surfaces.

The viewing orientation data is preferably provided periodically, eg at a selected rate, and preferably at a predefined rate, to a translation stage of the display controller, which then controls its translation operation. For this purpose, use the provided field of view orientation data, as appropriate. In a preferred embodiment, updated view orientation data is provided to the translation stage at a rate of several hundred hertz, such as 800 Hz.

View orientation data used by the conversion stage in generating the output transformed surface from the input surface may be provided to (and received by) the conversion stage in any suitable and desired manner. The viewing orientation data is preferably written to a suitable local storage (eg register or registers) of the display controller where it can be read and used by the conversion stage when generating the output transformed surface from the input surface.

In a preferred embodiment, the viewing orientation data is, for example, and preferably, the head position sensed from a suitable head position (head pose tracking) sensor of the virtual reality display headset for which the display controller is providing the image for display. It contains data (head pose tracking data).

For example, the sampled viewing orientation (eg, head position (pose)) data is preferably in any suitable manner and at a suitable rate (eg, at the same rate as being sampled by the associated head mounted display). to the conversion stage of the display controller. The transform stage may then use the provided head pose tracking (view orientation) information, as appropriate, to control its transform operation.

Thus, preferably, the viewing orientation data is suitably determined periodically, for example, and preferably by a virtual reality headset to which the display controller is coupled (and providing an output transformed surface for display). Contains sampled head pose tracking data.

The display controller may be integrated into the headset (head mounted display) itself or coupled to the headset via a wired or wireless connection.

Thus, in one particularly preferred embodiment, the method of the present invention is adapted for use by a conversion stage (e.g., and preferably, of a head mounted display in which the display controller is providing an output converted surface for the display). (using a suitable sensor) periodically sampling eye orientation data (e.g., head position data) and providing the sampled eye orientation data to a display controller (to a conversion stage of the display controller); and the display controller and/or data processing system are suitably configured to perform these steps), then the conversion stage converts the input surface to provide a transformed version of the input surface as the output transformed surface. It uses field of view orientation data.

The transform stage preferably bases itself on new view orientation data (head tracking data) at suitable intervals, such as at the beginning of each frame and/or after some number of lines of a given output transformed surface have been generated. It is configured to update the conversion operation of Preferably, the transformation stage is performed periodically based on the most recently provided viewing orientation (head tracking) information, and preferably each time a particular, preferably selected, number of lines of the transformed output surface are generated. , update its behavior. This can be done for each new line in the power transformed surface, or it can be done every few lines, eg 2 or 4 lines, in the power transformed surface.

The transform stage is operable to transform the input surface based on the received viewing orientation data in any suitable and desired manner. In one preferred embodiment, the transform stage is operative to apply a transform to the input surface based, in effect, on the viewing orientation data.

The transform that the transform stage applies to the input surface based on (substantially) the viewing orientation data can be any suitable and desired transform. In a preferred embodiment, the transformation is performed as if (e.g. compared to the generated (rendered) viewing orientation (viewing direction) with respect to the input surface) the viewing orientation (indicated) by the received viewing orientation data. direction) to transform the input surface to provide a representation of the input surface as viewed from

In one particularly preferred embodiment, the translation stage is configured to project a projection of the input surface based on the viewing orientation data, eg, and preferably, a projection based on and corresponding to a current viewing orientation as indicated by the viewing orientation data. works to generate

Transformations based on view orientation data can, and preferably include, for example, one or more of, and preferably all of, an x/y shift, scaling, and rotation. If desired, more complex transformations may be used.

In one preferred embodiment, the transformation based on the viewing orientation data includes (alone) a plane rotation of the input surface based on the viewing orientation data.

In one particularly preferred embodiment, the transform stage transforms the input surface based on (and taking into account) the viewing orientation as well as one or more other factors.

In this particularly preferred embodiment, the transform that the transform stage applies to the input surface also takes into account any distortion, eg, barrel distortion, caused by a lens or lenses through which the displayed output transformed surface to be viewed by the user passes. (Based on this).

Applicant has recognized at this point that the output frame displayed by the virtual reality headset is typically viewed through a lens, which lens typically imparts a geometric distortion, such as barrel distortion, to the viewed frame. It is therefore desirable to be able to “pre-distort” the displayed surface (frame) to compensate for the (geometric) distortion that the lens will produce.

Thus, in one particularly preferred embodiment, the transformation stage takes into account the (expected) (geometric) distortion from the lens or lenses through which the transformed output surface to be viewed passes, and the (expected) (geometric) distortion of that lens. It is operable to generate an output transformed surface from an input surface for compensation (consideration).

Thus, in one particularly preferred embodiment, the transformation stage is configured to have a specific, preferably selected, preferably predefined, (geometric) distortion of the lens (the lens distortion is caused by the lens through which the output transformed surface to be viewed by the user passes). It is operable to generate an output transformed surface from an input surface based on, and preferably based on, and corresponding to, the geometric distortion to be generated.

In a preferred embodiment, the transform that the transform stage applies to the input surface to create the output transformed surface is, in addition or instead (and preferably in addition to) the chromatic aberration ( distortion) (and is operable to try to correct for it).

Applicant has recognized at this point again that the lens through which the output frame displayed by the virtual reality headset to be viewed may also introduce chromatic aberrations into the frame being viewed. Accordingly, it is desirable to be able to "pre-calibrate" the displayed surface (frame) to compensate for any chromatic aberrations that the lens will introduce.

Thus, in one particularly preferred embodiment, the conversion stage takes into account the chromatic aberration in the viewing path of the (expected) output transformed surface (e.g. from the lens or lenses through which the transformed output surface to be viewed is passing); It is operable to generate an output transformed surface from an input surface to compensate for (account for) that (expected) chromatic aberration.

Thus, in one particularly preferred embodiment, the transformation stage is configured to have a specific, preferably selected, preferably predefined, chromatic aberration (the chromatic aberration should be based on the chromatic aberration that the output transformed surface to be viewed by the user would be produced by the lens through which it passes). and must correspond thereto, preferably based thereon and corresponding thereto) to generate an output transformed surface from an input surface.

In one particularly preferred embodiment, chromatic aberration correction is determined separately for the different color planes of the input surface. In one particularly preferred embodiment, individual chromatic aberration corrections are determined for each color plane of the input surface. Thus, in this case there will be respective red, green and blue (eg) chromatic aberration corrections determined by the translation stage and applied to the input surface.

Applicant has recognized at this point that any chromatic aberration introduced by the viewing path through which the output converted surface to be viewed will change depending on the color in question. Thus, instead of applying a single, common chromatic aberration correction to all color planes (although this can be done if desired), it is beneficial to determine individual chromatic aberration corrections individually for each color plane.

In a particularly preferred embodiment, if the input surface and the output transformed surface also contain alpha (transparency) data (alpha (transparency plane)), the determined for one of the color planes (preferably the green color plane) Chromatic aberration correction is also applied to (and used against) the alpha (transparency) plane (value).

The transform stage can produce an output transformed surface from an input surface in any suitable and desired manner.

The output transformed surface that the transform stage produces will be a frame (eg image) for display, and thus a corresponding data element (sampling position) on which each color data is defined (and provided) (eg e.g. pixels).

In one particularly preferred embodiment, the transform stage is such that, relative to a data element (sampling) (e.g. pixel) position in the output transformed surface, the data for that data element (sampling position) in the transformed output surface is It acts to determine the corresponding position (coordinate) on the input surface that should be taken. In other words, the transform stage preferably determines the position on the input surface to be moved to that data element (sampling position) on the output transformed surface by the transform that the transform stage applies to the input surface.

This is preferably done for a plurality and preferably for each data element (sampling position) in the output transformed surface it is desired to generate (for display).

The transform stage may determine which input surface locations should be sampled for a given transformed output surface data element (sampling) (eg, pixel) location in any suitable and desired manner.

In a particularly preferred embodiment, the transform stage is such that, for an output transformed surface data element (sampling) position x, y, the data for that data element (sampling position) in the transformed output surface is one or more of the following: Preferably determine the corresponding input surface position to be taken (to be taken) based on all: lens distortion correction configured such that a transform operation is applied to the input surface; a view orientation transform configured such that the transform operation is applied to the input surface; and chromatic aberration correction configured such that a transformation operation is applied to the input surface.

In a particularly preferred embodiment, the transform stage outputs, for each transformed surface sampling position x, y, the corresponding input surface position (coordinate) (x', y') based on the lens distortion the transform operation is configured to take into account. , and then modify the determined input surface position (coordinate) (x', y') based on at least the view orientation data (ie, to apply the necessary head tracking (view direction) transform).

More preferably, the determined input surface positions (coordinates) are also modified to account for chromatic aberrations when viewing the output transformed surface (to apply chromatic aberration correction). This is preferably done prior to modifying the input surface location determined based on the viewing orientation data.

Correspondingly, in one particularly preferred embodiment, the conversion stage outputs, relative to the sampling position (coordinates) on the output transformed surface to be output from the conversion stage, the output transformed surface sampling position (coordinates) based on the defined lens distortion. a coordinate interpolator stage operable to determine a corresponding input surface position (coordinate) for ; a chromatic aberration correction stage operable to correct input surface positions (coordinates) to account for chromatic aberration when viewing the output transformed surface on its display; and a viewing orientation conversion stage operable to modify the input surface location (coordinates) based on the received viewing orientation data (eg, indicating a current head position of a viewer who will view the output surface).

This way of transforming an input surface to give an output transformed surface by determining the position(s) (coordinates) on the input surface that should be used for a given sampling location (pixel) on the output transformed surface is essentially novel. and beneficial, and may not be intended solely for use in a display for a display device. For example, it may be possible to provide this conversion stage as a stand-alone model that can be included in the entire data processing system (system on chip) and not necessarily just within the display controller.

Accordingly, according to a further aspect of the present invention there is provided an apparatus for transforming an input surface to provide an output transformed surface based on received viewing orientation data, the apparatus comprising:

a coordinate interpolator stage operable to determine, for data element positions on an output transformed surface to be output from the device, a corresponding data position on the input surface based on the defined lens distortion;

a chromatic aberration correction stage operable to correct the input surface location determined by the coordinate interpolator stage to account for the chromatic aberration when viewing the output transformed surface to provide at least one corrected input surface location; and

further comprising at least one corrected input surface position determined by the chromatic aberration correction stage based on the received viewing orientation data to provide an output input surface position that is sampled to provide data used for data elements of the output transformed surface; A field of vision orientation conversion stage operable to correct

includes

Correspondingly, according to a further aspect of the present invention, there is provided a method of transforming an input surface to provide an output transformed surface based on received viewing orientation data, the method comprising:

for data element positions on the output transformed surface to be output, determining a corresponding position on the input surface based on the defined lens distortion;

when viewing the output transformed surface to provide at least one corrected input surface position, modifying the determined input surface position to account for chromatic aberration; and

further modifying the at least one corrected input surface location based on the received viewing orientation data to provide an output location on the input surface that is sampled to provide data used for data elements of the output transformed surface;

includes

As will be appreciated by those skilled in the art, these aspects of the invention may include any one or more or all of the preferred and optional features of the invention described herein, as appropriate. Yes, and preferably includes.

As discussed above, in one preferred embodiment, the chromatic aberration correction stage is operable to determine individual chromatic aberrations for each different color plane. Thus, in one particularly preferred embodiment, the process (as discussed above) performs an initial input to the output transformed surface data element (sampling position) that is being generated, for example, and preferably based on the lens distortion. Although operable to determine the surface position, the chromatic aberration correction stage preferably comprises a plurality (eg, and preferably three, one in each color plane) corrected input surface positions (coordinates) (each corresponding chromatic aberration). explain) is determined.

In this way, the subsequent modification of the input surface location based on the viewing orientation must, therefore, be applied to each input surface location generated (e.g. to each different color plane input surface location), and preferably applied do.

Thus, in this case, for a given output transformed surface sampling location, a plurality (eg, three) locations sampled on the input surface, for example, and preferably, each to be generated for the output transformed surface. There will be (and preferably is) one input surface location for a different color plane (color value) of .

Once a location or locations on the input surface where data is to be used for data elements (sampling locations) in the output transformed surface are determined, in one particularly preferred embodiment, the input surface is sampled at the determined location or locations, so that the output transformed surface is Provides data values to be used for data points (sampling locations) on the surface.

The input surface may be sampled in any suitable and desired manner in this regard.

In one particularly preferred embodiment, the translation stage provides data for a corresponding input surface location determined from data values defined for a set of multiple (eg four) input surface sampling locations surrounding that input surface location. By interpolating values, it is operable to determine data values for data elements (sampling positions) in the output transformed surface.

Thus, in one particularly preferred embodiment, the conversion stage is configured from a defined sampling position on the input surface to provide data about the input surface position (coordinate) to be used for the output transformed surface data element (sampling) position (coordinate). It is operable to interpolate the data of

Accordingly, in one particularly preferred embodiment, a transform stage is an interpolator operable to interpolate a plurality of input surface sampling position values to provide interpolated sampling position values for use with data elements (sampling positions) in the output transformed surface. contains the stage

Interpolation of the input surface sampling position data may be performed in any suitable and desired manner. In one preferred embodiment, bilinear interpolation is used, but other schemes may be used if desired.

In this way, if different input surface positions (coordinates) are determined for different color planes (taking into account chromatic aberrations), in a preferred embodiment, a separate data set is thus generated for each color surface corresponding to that color surface. Based on the input surface position for the output transformed surface data element (sampling position) is interpolated. Thus, for example, an RGB color data set for an output transformed surface data element (sampling position) preferably has a red value determined from one ("red") input surface position (coordinate), and another ("red") will contain a green value determined from the input surface location (coordinates) of "green", and a blue value determined from the third ("blue") input surface location (coordinates).

These individually determined color plane (eg red, green and blue) values are then preferably used together to provide color data for the output transformed surface data element (sampling position) in question.

If the alpha (transparency) value is also determined, as discussed above, it is preferably determined from the input surface location (coordinates) used to determine the green color value.

In one particularly preferred embodiment, the transform stage reads as input one or more sampling locations (eg pixels) from the input surface and produces output sampling locations (eg pixels) of the output transformed surface such sampling locations. It will be appreciated from above that it works by using In other words, the transform stage preferably outputs data values for corresponding sampling positions (eg pixels) on the transformed surface from data values for sampling positions (eg pixels) on the input surface. By generating, the output operates to create a transformed surface.

(As understood by those of skill in the art, defined sampling (data) locations (data elements) at the input surface (and at the output transformed surface) may correspond to pixels on the display (and (preferably corresponding), but it need not be. For example, if the input surface and/or the output transformed surface are subjected to some form of downsampling, the surface is sampled (data ) there will not be a one-to-one mapping of positions, but rather sets of multiple data (sampling) positions (data elements) on the input surface and/or output transformed surface corresponding to each pixel of the display.)

Thus, in one preferred embodiment, the transform stage is configured for each sampling position (data element), preferably for a plurality of sampling positions (data element), and preferably for each required sampling position (data element) for the output transformed surface. Operates on a sampling location (data element) and, relative to that output transformed surface sampling location, one or more input surface sampling locations (and preferably a plurality of input surface sampling locations) to be used to generate that output transformed surface sampling location. set), and then use these determined input surface sampling locations or locations to generate output transformed surface sampling locations (data elements).

Generating each sampling location (data element) from the output transformed surface in this way makes it possible to generate the output transformed surface on a sampling location (by data element) basis (e.g., and preferably , as a sequence or stream of output sampling locations (data elements). Because this effectively ensures that the output-transformed surface behaves in a way consistent with the way the display controller and its display processing pipeline would normally handle a surface for display (i.e. the sampling location (pixels) of In contrast to a block, as a sequence or stream of sampling locations (data elements) (eg pixels), this is advantageous.

In fact, a particular advantage of these embodiments of the present invention is that they enable the creation of a power transformed surface as a stream of sampling locations of that surface, whereby the power transformed surface is being displayed, in "real time", display It enables the ability to perform temporal warp rendering on the display controller as part of the processing pipeline.

Thus, in one particularly preferred embodiment, the transform stage is operative to generate an output transformed surface on a sampling location-by-sampling (eg, pixel-by-pixel) basis.

Thus, in one preferred embodiment, the transform stage, relative to the sampling position (data element) in the output transformed surface, the sampling position or positions (data element or elements), then use this input surface sampling location or locations to create an output transformed surface sampling location, and then use the next (or second) output transformed surface sampling location (data element), determine the sampling location or locations on the input surface to be used to generate the next (second) output transformed surface sampling location, and then determine this determined input to generate the output transformed surface sampling location. It uses a surface sampling location or locations and continues to operate similarly.

Thus, in one particularly preferred embodiment, the output of the transform stage is a stream of output transformed surface sampling positions (sampling position values).

The transform stage preferably creates the output transformed surface from the input surface on a line-by-line (raster line-by-raster) basis, as it is in a suitable format for providing the output transformed surface to the display. Accordingly, the transform stage preferably produces the output transformed surface as a sequence of rows of sampling locations (data elements) of the output transformed surface, each row having a width for the output transformed surface and an output transformed surface It has a length corresponding to the desired number of sampling locations across the vertical height of one sampling location on the surface.

Applicant points out in this respect that it may be the case that the transform stage generates the surface as a row sequence of output transformed surfaces, for example as a horizontal sequence of rows, for example, but to be compensated for by the transform stage, for example. For example, because of barrel distortion, it may be the case that a given row of output transformed surfaces solely uses the input surface sampling location from the corresponding horizontal row in the input surface, in fact this usually will not be the case. Rather, rows of the output transformed surface will typically be formed from sampling locations taken from a curve extending across the input surface.

To take this into account, and then to be able to generate an output transformed surface as a contiguous sequence of sampling positions for a row of transformed output surfaces, in one particularly preferred embodiment, a transformation stage is currently created and Identify the regions (and preferably corresponding two-dimensional blocks (areas)) of the input surface from which data will be needed to generate rows of transformed output surfaces that are located, and the corresponding output transformed surface sampling location or locations from which to be created. is operable to cause the input stage of the display controller to load the required input surface area (block) into the input buffer or buffers of the display controller, for example a local buffer or buffers, to be available for use by the conversion stage when .

The block (region) of the input surface loaded in this respect may be any suitable and desired block (region) of the input surface. Each block preferably contains a (two-dimensional) array of defined sampling (data) locations (data elements) of the input surface, extending in each axial direction by a plurality of sampling locations (data elements). The blocks are preferably rectangular and preferably square. A block may contain, for example, 4x4, 8x8 or 16x16 sampling locations (data elements) of the input surface, respectively.

In one particularly preferred embodiment, this means that the transform stage will sample the output transformed surface sampling locations (data elements) to identify blocks (areas) of the input surface from which data will be needed to generate the output transformed surface sampling locations. It uses the integer part of the input surface position, and then the appropriate block of the input surface into the display controller's local buffer from which the conversion stage reads the desired input surface sampling position to create an output transformed surface sampling position. This is done by indicating the block position (e.g. an integer value) to the input stage of the display controller to cause the fetch.

In one preferred embodiment, the operation, eg the input stage, first determines if the required input surface block already exists in the local buffer, and then if it is determined that the input surface block does not already exist in the local buffer. , it is configured only to fetch its input surface block from memory into a local buffer. Correspondingly, in one preferred embodiment, when a new input surface block is to be fetched into the local buffer, the input surface block already stored in the local buffer is, if necessary, preferably a suitable block, such as the most recently used replacement process. It is evicted (replaced by a new block) using the replacement mechanism.

The output transformed surface produced by the conversion stage may be provided to the output stage as an output surface for a display to present as an output surface for display to a display in any suitable and desired manner.

In a preferred embodiment, the output of the transform stage (eg, a stream of output transformed surface sampling locations (data elements)) is output suitable for the rest of the display processing pipeline for processing. For example, and preferably, a further processing stage of the display controller, for example, and preferably a composition stage, for any desired further processing prior to display, and in a preferred embodiment provided do.

Thus, in one preferred embodiment, providing the output transformed surface to an output stage to serve as an output surface for a display is equivalent to providing the output transformed surface to a further processing stage of the display controller and then to the display. and providing the thus processed output transformed surface to an output stage (or, for example, to a further processing stage) to provide as an output surface for display. The output transformed surface may, if desired, be provided to a plurality of processing stages prior to being provided to the output stage.

In one particularly preferred embodiment, the output of the translation stage (e.g., the sampling position for the output transformed surface produced by the translation stage) is transferred from the translation stage to the subsequent stage of the display controller (e.g., the output A buffer between stages and/or one or more processing stages prior to the output stage), eg a suitable buffer (eg a delay hiding buffer such as a FIFO) serving as a latency hiding buffer.

The output stage of the display controller of the present invention may be any suitable output stage operable to provide an output surface for display to a display, for example, to cause the output surface for display to be displayed on the display. The output stage preferably includes a processing pipeline that performs the necessary display processing operations on the output surface to be displayed. The output stage preferably includes, for display, suitable timing control functions (eg it is configured to transmit pixel data to the display with suitable horizontal and vertical blanking periods).

The transformation stage (and each stage thereof, eg, coordinate interpolator stage, chromatic aberration correction stage, and field orientation transformation stage) may be implemented in any suitable and desired manner. For example, they may be provided using suitable programmable processing circuitry that can be programmed to operate in a desired manner. They may all be implemented in this respect by the same programmable processing circuit, each programmed to perform a different operation, or there may be separate programmable stages each programmable to operate in any desired manner.

However, in one particularly preferred embodiment, the conversion stages (and each individual stage thereof) are not provided using programmable processing circuitry, but instead fixed function stages (fixed function) configured to perform the necessary processing operations. function processing circuit). Thus, in one particularly preferred embodiment, the conversion stage (and each of its corresponding individual stages) is configured to perform the necessary operations (and executes (via a programmable processing circuit) a software program to perform the required operations). It is provided in the form of fixed function hardware units (circuits) that do not execute).

Applicant believes in this respect that providing a fixed function circuit operable to perform transformations of the input surface (eg determining various input surface positions) is a programmable circuit that may have variable delays and delays in use, for example. It has been recognized that, in contrast to the manner in which operations are implemented by running a suitable program using an appropriate program, a translation stage will enable it to perform its operations at a known and reliable (eg, fixed) rate. Thus, this further enables real-time performance of the operations of the present invention at the display controller (in the display processing pipeline of the display controller).

In one particularly preferred embodiment, where the fixed function circuit is used to perform the operation of the conversion stage, the fixed function circuit is arranged to have limited configurability during use. Particularly advantageously, it is possible to change one or more control parameters relative to the operation performed by the circuit in order to allow some variation in the operation of the fixed function circuit and configuration of the operation of the fixed function circuit. Thus, in one preferred embodiment, the operation of the translation stage is implemented using a fixed function processing circuit, but is configurable by setting one or more specific control parameters, preferably selected control parameters, for the circuit during use.

For example, where a translation stage is operable to transform an input surface to compensate for potential lens (geometric) distortion, for example one stage of the translation stage, the circuitry may be used in a general "lens distortion" compensation process. However, the exact parameters of the process being performed are set using one or more control parameters that can be set during use. For example, lens distortion can be controlled and expressed using a 2D grid with fiducial points approximating the lens distortion.

Similarly, if the conversion stages are operable to transform the input surface to compensate for (correct) potential chromatic aberrations, for example one stage of the conversion stages, the circuit may perform the normal "chromatic aberration" compensation (correction) process. However, the exact parameters of the process being performed are set using one or more control parameters that can be set during use.

Equally, where the transform stage is operable to transform the input surface based on the received viewing orientation, for example one stage of the transforming stage, the circuitry may be configured to perform a general “viewing orientation” conversion process. However, the exact parameters of the process being performed are set using one or more control parameters (eg Euler angles or quaternions) that can be set in use.

In these approaches, the control parameters for the conversion stage can be set as desired. For example, if the display controller is only configured to operate with a particular given virtual reality head-mounted display (eg, provided as part of a fixed virtual reality head-mounted display system), the translation stage for The control parameters can be set as part of the initial setup process and not to change during use (i.e., the translation stage is simply preset to perform the appropriate operation for the display in question, e.g., lens distortion correction, etc. can be configured).

In a preferred embodiment, the control parameters for the translation stage are set on-the-fly, for example, and preferably by an application providing an input surface to a display controller for the display and/or by a driver for the display controller. It can be, and is preferably set.

This then allows, for example, the display controller to be used with different headsets, for which, for example, lens distortion correction, etc., the current head-mounted display with which the display controller is being used (to which the display controller is connected) based on , for example by a driver, to be set during use.

Any other operations such as chromatic aberration correction that may be applied by the translation stage may be similarly configured (i.e. pre-configured (pre-stored) for "fixed" operation, or for example and preferably a display controller may be configurable during use, for example by a driver, to account for different usage situations for

In addition to the input stages, output stages, and translation stages discussed above, the display controller of the present invention may include any one, one or more, or all of the other processing stages and elements that a display controller may suitably include.

As discussed above, in one preferred embodiment, the display controller includes a processing stage or stages operable to process a surface or surfaces, for example to create an output surface.

The processing stage or stages preferably include: compositing (two or more) input surfaces to create a composite output surface; decoding (eg, decompressing) an input surface, eg, to produce one or more decoded (eg, decompressed) input surfaces; rotating the input surface, eg, to create one or more rotated input surfaces; and to perform one or more, and preferably all, of, for example, scaling (eg, upscaling and/or downscaling) one or more surfaces to create one or more scaled surfaces. possible.

These operations may be performed by the same processing stage, or if desired, there may be, for example, each separate synthesis stage, decoding (decompression) stage, rotation stage and/or scaling stage. Also, if desired, it would be possible to provide some or all of this functionality at the input stage.

The display controller preferably applies one or more processing operations to one or more input surfaces, as appropriate, prior to presenting the one or more processed input surfaces to, for example, a translation stage, scaling stage, and/or compositing stage or the like. and one or more layer pipelines operable to perform. In cases where a display controller can handle multiple input layers, there may be multiple layer pipelines such as a video layer pipeline or pipelines, a graphics layer pipeline, and the like. These layer pipelines may be operable to provide pixel processing functions such as, for example, pixel unpacking, color conversion, (inverse) gamma correction, and the like.

Additionally, the display controller may include a post-processing pipeline operable to perform one or more processing operations on one or more surfaces, eg, to create a post-processed surface. This post-processing may include, for example, color conversion, dithering, and/or gamma correction.

In one preferred embodiment, the display controller further comprises a write-out stage operable to write the output surface to external memory. This allows the display controller to (optionally) write the output surface, eg to an external memory (eg frame buffer), eg, and preferably, at the same time that the output surface is being displayed on the display. will make it

In this way, in one preferred embodiment, the display controller is operative to display the thus output transformed surface and write it to external memory (while it is being generated and presented by the transformation stage). This can be useful, for example, where an output transformed (time warped) surface may be required to be created by applying a different set of values to a previous ("baseline") output transformed surface. In this case, the write stage of the display controller can then be used to store, eg, a "reference" output transformed surface in memory so that it is available for use in creating future output transformed surfaces. .

Of course, other methods are also possible.

The various stages of the display controller may be, for example, in the form of one or more fixed function units (hardware) (ie dedicated to one or more functions that cannot be changed), or programmed, for example, to perform desired operations. One or more programmable processing stages using programmable circuitry, which can be implemented as desired. There can be both fixed function stages and programmable stages.

One or more of the various stages of the display controller may be provided as separate circuit elements. Additionally or alternatively, some or all of the stages may be formed at least in part from shared circuitry.

It may also be possible for the display controller to include, for example, two display processing cores, one or more or all of the cores being configured in the manner of the present invention, if desired.

The display with which the display controller of the present invention is used can be any suitable and desired display (display panel), such as, for example, a screen. This may include local displays (screens) of the entire data processing system (of the device) and/or external displays. If desired, there may be more than one display output.

In one particularly preferred embodiment, the display with which the display controller is used comprises a virtual reality head mounted display. The display thus preferably comprises a display panel for displaying to the user the output transformed surface produced in the manner of the present invention, and a lens or lenses through which the displayed transformed output surface to be viewed by the user passes.

Correspondingly, the display preferably generates eye tracking information, preferably periodically, based on the current and/or relative position of the display, when transforming the input surface to provide an output transformed surface for display. Associated a viewing orientation determination (eg, head tracking) sensor operable to periodically provide its viewing orientation data to the display controller for use (to the conversion stage of the display controller).

In one preferred embodiment, the display controller of the present invention forms part of a data processing system. Accordingly, according to another aspect of the present invention, a data processing system comprising a display controller according to the present invention is provided.

Further, the data processing system may include, and preferably includes, one or more, and preferably all, of a central processing unit, a graphics processing unit, a video processor (codec), a system bus, and a memory controller.

The display controller and/or data processing system may be, and preferably is, configured to communicate with one or more of the following (and the present invention extends in a manner that also includes one or more of the following): External memory (e.g. eg, through a memory controller), one or more local displays, and/or one or more external displays. The external memory preferably includes the main memory (eg shared with a central processing unit (CPU)) of the entire data processing system.

Accordingly, in some embodiments, the display controller and/or data processing system may include one or more memories and/or memory that stores software for storing one or more data described herein and/or performing processes described herein. or a memory device, and/or in communication with it. The display controller and/or data processing system may also communicate with and/or include a host microprocessor and/or communicate with a display for displaying images based on data generated by the display controller and/or may include this.

Correspondingly, according to a further aspect of the invention,

main memory;

display;

one or more processing units operable to create an input surface for display and to store the input surface in main memory; and

A data processing system is provided that includes a display controller, the display controller comprising:

an input stage operable to read an input surface stored in main memory;

an output stage operable to provide an output surface for display on a display; and

transforming the input surface read by the input stage based on the received viewing orientation data to provide a transformed version of the input surface as an output transformed surface;

and a conversion stage operable to provide an output transformed surface to the output stage for providing as an output surface for display to a display.

As will be appreciated by those skilled in the art, these aspects and embodiments of the invention may include one or more and preferably all of the preferred and optional features of the invention described herein. may and preferably include.

Thus, for example, the display controller preferably further comprises one or more local buffers, the input stage of which preferably goes from main memory to the local buffer or buffers of the display controller (for processing by the subsequent conversion stage). ) to fetch data of the input surface to be processed by the display controller.

In use of the display controller and data processing system of the present invention, one or more input surfaces will be created and stored in memory, for example and preferably by GPUs, CPUs and/or video codecs, etc. This input surface will then be processed by the display controller to provide an output surface for display to the display.

While the present invention has been described above with particular reference to the creation of a single output transformed surface from an input surface, as will be appreciated by those skilled in the art, at least in preferred embodiments of the present invention, a plurality of generated There will be an input surface, representing successive frames of the frame sequence to be displayed to the user. The conversion stage of the display controller will thus preferably operate to provide a sequence of a plurality of output converted surfaces for display. Thus, in one particularly preferred embodiment, operations in the manner of the present invention are used to generate a sequence of a plurality of output transformed surfaces for display to a user. Correspondingly, the operation in the inventive scheme is preferably repeated for a plurality of output frames to be displayed, for example, and preferably for a sequence of frames to be displayed.

Additionally, creating the power transformed surface may accordingly and correspondingly include creating a sequence of “left” and “right” power transformed surfaces to be displayed to the user's left and right eyes, respectively. Each pair of “left” and “right” output transformed surfaces may be generated from a common input surface or from corresponding “left” and “right” input surfaces, as desired.

The invention may be implemented in any suitable system, such as a suitably configured microprocessor based system. In one embodiment, the invention may be implemented in a computer and/or microprocessor based system.

The present invention is preferably implemented in a virtual reality display device, such as, and preferably a virtual reality headset. Accordingly, according to another aspect of the present invention there is provided a virtual reality display device comprising a display controller and/or data processing system of any one or more of the aspects and embodiments of the present invention. Correspondingly, according to another aspect of the present invention, there is provided a method of operating a virtual reality display device comprising operating the virtual reality display device in the manner of any one or more of the aspects and embodiments of the present invention.

The various functions of the present invention may be performed in any suitable and desired manner. For example, the functionality of the present invention may be implemented in hardware or software, as desired. Thus, for example, unless otherwise indicated, the various functional elements, stages and "means" of the present invention may include suitable dedicated hardware elements (processing circuits) and/or programmable hardware elements (processing circuits) that can be programmed to operate in a desired manner. circuitry), a suitable processor or processors, controller or controllers, functional units, circuits, processing logic, microprocessor devices, etc. operable to perform various functions and the like.

Further, it will be noted that various functions and the like of the present invention may be copied and/or executed in parallel on a given processor, as will be appreciated by those skilled in the art. Equally, the various processing stages may share processing circuitry, etc., if desired.

Additionally, any one or more or all of the processing stages of the present invention may be, for example, in the form of one or more fixed function units (hardware) (processing circuits) and/or programmable processing circuits that may be programmed to perform desired operations. In the form of, may be embodied as a processing stage circuit. Equally, any one or more of the processing stages and processing stage circuits of the present invention may be provided as discrete circuit elements on any one or more of the other processing stages or processing stage circuits, and/or of processing stages and processing stage circuits. Any one or more or all of them may be formed at least in part with shared processing circuitry.

Further, all of the described aspects and embodiments of the present invention may, as appropriate, include any one or more or all of the preferred and optional features of the present invention described herein, and in one embodiment It will be understood by one of ordinary skill in the art to include.

The method according to the invention can be implemented at least partly using software, eg a computer program. Accordingly, when considered from further embodiments, it will be seen that the present invention provides computer software specifically adapted to perform the methods described herein when installed on a data processor, the computer program elements comprising the program elements A computer program comprising computer software code portions for performing the methods described herein when executed on a data processor, wherein the computer program is adapted to perform the method or all steps of the methods described herein when the program is executed on a data processing system. contains software code; The data processor may be a microprocessor system, a programmable field programmable gate array (FPGA), or the like.

The invention also extends to a computer software carrier containing software which, when used to operate a microprocessor containing a display controller or data processor, causes said controller or system, together with said data processor, to execute the steps of the method of the invention. . Such a computer software carrier may be a physical storage medium such as a ROM chip, CD ROM, RAM, flash memory or disk, or may be a signal such as an electronic signal over a wire, an optical signal or a radio signal, for example to a satellite or the like. .

It will be further appreciated that not all steps of the methods of the present invention need be executed by computer software, and thus, from a broader embodiment, the present invention provides computer software and methods for executing at least one of the steps of the methods disclosed herein. Provide such software installed on a computer software carrier.

Accordingly, the present invention may be suitably embodied as a computer program product for use with a computer system. Such an implementation may include a set of computer readable instructions affixed to a tangible, non-transitory medium such as a computer readable medium, for example a diskette, CD-ROM, ROM, RAM, flash memory or hard disk. It may also be through a modem or other interface device, over a tangible medium including, but not limited to, optical or analog communication lines, or by intangible use of radio technologies, including but not limited to microwave, infrared, or other transmission technologies. , a set of computer readable instructions transmittable to a computer system.

Those skilled in the art will understand that such computer readable instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Further, such instructions may be stored using any current or future memory technology including but not limited to semiconductor, magnetic or optical, or any present or future including but not limited to optical, infrared or microwave. It can be transmitted using the communication technology of. Such computer program products may be distributed as removable media with attached printed or electronic documentation, e.g. shrink-wrapped software, or pre-loaded into a computer system, e.g., in a system ROM or fixed disk, or , or from a server or bulletin board via a network, eg, the Internet or the World Wide Web.

Various embodiments of the present invention will now be described by way of example only and with reference to the accompanying drawings in which:
1 shows an exemplary data processing system;
2 and 3 illustrate the process of “time warp” rendering in a head mounted virtual reality display system;
4 schematically illustrates a display controller operable in accordance with one embodiment of the present invention;
Fig. 5 shows the conversion stage of the display controller of Fig. 4 in more detail;
Fig. 6 is a flow chart illustrating operation of the conversion stage of the display controller of Fig. 4 in more detail;
Figure 7 schematically illustrates the creation of an output transformed surface for display in one embodiment of the present invention;
8 schematically illustrates an exemplary virtual reality head mounted display headset;
9 schematically illustrates the provision of a frame for display in a data processing system;
Figure 10 schematically illustrates the provision of a frame for display in a data processing system in one embodiment of the present invention;
Figure 11 schematically illustrates in more detail the provision of a frame for display in a data processing system in one embodiment of the present invention; and,
12 schematically illustrates an example of a lens distortion correction parameter that may be used in one embodiment of the present invention.
Like reference numbers are used for like elements throughout the drawings where appropriate.

A number of preferred embodiments of the present invention will now be described.

The present invention and its embodiments relate to a process for displaying frames to a user in a virtual reality display system, particularly in a head mounted virtual reality display system.

Such a system may be configured as shown in Figure 1 (and as described above), wherein the display 4 of the system is, inter alia, a display for displaying frames to be viewed by a user wearing a head mounted display. the screen or screens (panel or panels), one or more lenses in the viewing path between the user's eyes and the display screen, and, in use (while an image is being displayed on the display to the user) the user's head ( and/or a suitable head mounted display that seeks one or more sensors for tracking the position (pose) of its field of view (line of sight) direction.

8 schematically depicts an exemplary virtual reality head mounted display 85 . As shown in FIG. 8 , the head mounted display 85 includes a suitable display mount 86 having, for example, one or more head pose tracking sensors to which a display screen (panel) 87 is mounted. A pair of lenses 88 are mounted in a lens mount 89 in the viewing path of the display screen 87. Finally, there is a suitable fit 95 for the user to wear the headset.

In the system shown in Figure 1, the display controller 5 will operate to present an image suitable to the display 4 for viewing by a user. The display controller can be coupled to the display 4 in a wired or wireless manner as desired.

Images displayed on the head mounted display 4 are generated, for example, by a graphics processor (GPU) in response to a request for such rendering from an application 10 running on a host processor (CPU) 7 of the entire data processing system. ) (2) and will store these frames in main memory (3). The display controller 5 will then read the frames from the memory 3 as an input surface and properly provide these frames to the display 4 for display to the user.

In this embodiment, and in accordance with the present invention, the display controller 5 performs a so-called "time warp" on the frames stored in the memory 3 before providing these frames to the display 4 for display to the user. )" processing is operable.

10 and 11 illustrate this.

As shown in FIG. 10 , a display controller 93 is operable to perform temporal warp processing of a rendered frame 92 generated by a graphics processor (CPU) 91 for presentation to a display panel 94 . A time warp module (transformation stage) 100 is included.

Fig. 11 shows the operation of the display controller when performing time warp processing. As shown in Figure 11, the rendered frame 92 to be displayed is first fetched from memory (step 110) and then loaded into a suitable buffer, such as the line buffer of the display controller 93 (step 111). ).

It is then determined whether temporal warp processing is enabled (step 112). (If time warp processing is not enabled, the loaded input frame is presented to display 94 for display in the usual way.)

If time warp processing is enabled, the fetched input frames are processed by the display controller's time warp module (conversion stage) based on the appropriate time warp parameters 114 provided from the host processor 90 to determine the fit of the frame to be displayed. provides a "timewarped" version (step 113). The "time warped" frames are then provided in the usual manner to the rest of the display processing pipeline (to panel signal generator 115) for presentation to display 94 for display.

Now, the configuration of the display controller 5 for performing the "time warp" operation in this embodiment and the process of this operation will be described in more detail.

4 schematically shows the display controller 5 in more detail. In FIG. 4 , boxes represent functional units of the display controller, and lines with arrow symbols represent connections between various functional units.

As shown in Fig. 4, the display controller 5 comprises a memory read subsystem 41 comprising, inter alia, a read controller in the form of a Direct Memory Access (DMA) read controller. The read controller is configured to read one or more input surfaces from one or more frame buffers in main memory 3 (not shown in FIG. 4) via memory bus 9.

Memory read subsystem 41 includes one or more real-time first-in-first-out (FIFO) interfaces used to locally buffer one or more input surfaces as they are read from memory, e.g., for latency hiding purposes. ) module.

In this embodiment, memory read subsystem 41 is configured to present (read) up to three different input surfaces for use as input layers used to generate synthesized output frames. The three input layers are, for example, one or more video layers generated by a video processor (codec) 1 and one or more graphic windows, eg generated by a graphics processing unit (GPU) 2. A graphic layer may be included. Figure 4 thus shows the display controller 5 comprising three layer pipelines 42, 43, 44 which will respectively receive data from the input surface to be used as the display layer. Any or all input surfaces received by the layer pipeline may have been subjected to decoding by the decoder and/or rotation by the rotation unit, if desired.

Each layer pipeline 42, 43, 44 performs pixel unpacking from received data words, color conversion (e.g., YUV to RGB conversion), and inverse gamma or inverse sRGB (inverse sRGB) data. sRGB) perform appropriate operations on the received surface.

Although the embodiment of Figure 4 illustrates the use of a three layer pipeline (and thus up to three input layers), depending on the application in question (and also subject to any silicon area limitations, etc.), the present invention may It will be appreciated that a number of layer pipelines may be provided and used.

The layer processing pipelines 42, 43, 44 take the data of the input surface read by the memory reading subsystem 41 and produce from that data output surface, e.g., for display, the display controller 5 serves as the processing stage for

The display controller 5 may receive input from the layer pipelines 42, 43, 44 and construct the received input layers to produce a synthesized output surface, eg by suitable alpha blending operations or the like. and a synthesizing unit (display synthesizer) 46 operative to:

The synthesized output frames from compositing unit 46 are further sent to display processing (pre-processing) pipeline 47 and/or memory write subsystem 48 for display, as desired.

Display pipeline 47 is configured to selectively perform any processing operation(s) on the synthesized output surface (frame) and then transmit the (processed) synthesized output frame for appropriate display on an associated display. It consists of

Display processing pipeline 47 may include, for example, a color conversion stage operable to apply a color transformation to the synthesized output frame, a dithering stage operable to apply dithering to the synthesized output frame, and/or gamma correction to the synthesized output frame. and a gamma correction stage operable to perform on the output frame.

Display processing pipeline 47 also includes suitable display timing functions. Accordingly, display processing pipeline 47 may be configured to transmit pixel data to display output 49, for example, with suitable horizontal and vertical blanking periods. For example, horizontal and vertical synchronization pulses (HSYNC, VSYNC) may be generated with the DATAEN signal asserted in a non-blanking period. In the blanking period, DATAEN is de-asserted and no data is sent to the display (there are four blanking periods: horizontal front porch - before the HSYNC pulse, horizontal back porch back porch - after HSYNC pulse, vertical front porch - before VSYNC pulse, vertical back porch - after VSYNC pulse).

Also, if desired, it would be possible to use other display timing and data (pixel) packing schemes such as MIPI DPI, HDMI, Display Port, etc.

Display output 49 may interface with, for example, a local display of the data processing system (eg, a mobile device, smartphone, tablet, etc. of which the data processing system is a part).

Display processing pipeline 47 and display output control interface 49 thus serve as an output stage for display controller 5 to provide an output surface for display on display 4 .

The memory writing subsystem 48 of the display controller 5 is operable to write the received surface, for example created by the composite unit 46, to the external memory 3 via a memory bus. This then allows the display controller 5 to not only provide output frames for display, but also to write these output frames into main memory if desired. To enable this operation, memory write subsystem 48 includes a DMA write controller. In this embodiment, it also includes a FIFO suitable to serve as a latency hiding buffer.

In addition, the display controller 5 controls data flow operable to direct data flow through the display controller, i.e. to provide input layers, synthesized output surfaces, etc. to suitable units for processing as shown in FIG. module 45. In this embodiment, the data flow controller 45 is, for example and preferably, running on a host processor (e.g. CPU 7) of the overall data processing system of which the display controller 5 is a part. It operates under suitable software control from the driver 11 for the display controller. The driver may generate appropriate commands for the data flow controller 45 and program the control registers of the display controller 5 in response to commands and data for display processing received, for example, from an application running on the host processor. can

Of course, other approaches are possible in this respect.

As discussed above, when the display controller 5 provides an output frame for display, it has been generated, for example, by the video codec 1 and/or the GPU 2, and the main memory 3 reads the data of one or more input surfaces stored in respective frame buffers within, serves as an input layer in its output surface creation process, processes the input surface data (e.g., by compositing it into an output frame), displays In order to provide (synthesized) output frames to the display 4 for display via the processing pipeline 47, for example, the data of one or more input surfaces stored in respective frame buffers in the main memory 3 are sent. will read

This process is repeated at a rate of, for example, 30 or 60 frames per second for each frame that needs to be displayed.

Since this display processing is a real-time operation, it is necessary for the display controller 5 to regularly pass the pixel data to be displayed to the display 4 (to the display output) in each clock cycle that triggers the display output from the display controller. there is

In addition to the functions discussed above, in this embodiment, the display controller 5 has a scaling function (scaler) in each of its input layer pipelines 42, 43, 44 and a display processing pipeline 47 in its output. ) (not shown).

As shown in FIG. 4 , the display controller 5 of this embodiment further comprises a conversion stage 50 in the form of a time warp module 50 having an associated input buffer 51 and an associated output buffer 52 . The time warp module 50 receives its input via an input buffer 51 to provide a suitable "time warped" version of the input surface, and then provides that input to the display via an output buffer 52 for display. It is operable to transform the surface.

The required input surface data directs the memory read subsystem 41 to read the appropriate data for the input surface from main memory and provide it to the layer pipeline for presentation to the input buffer 51 by the data flow control module 45. It is loaded into the input buffer 51 by the controlling driver.

As shown in FIG. 4 , data flow control module 45 transfers input surface data processed by layer pipelines 42 , 43 and/or 44 to an input buffer for processing by time warp module 50 . 51, and then properly direct data from the output buffer 52 to the composite unit 46 and thereafter to the display processing pipeline 47 for display. It can also be controlled and configured as appropriate by the driver.

Data flow control module 45 may also direct the data of the time warp output transformed surface to memory write subsystem 48 for writing to external memory, if desired. This may be suitable where subsequent frames for display are generated based on differences from previous frames.

Although input buffer 51 and output buffer 52 are shown as separate buffers in FIG. 4 , these buffers may also be present in display controller 5 if desired, such as buffers that may be present in a layer pipeline, for example. It can be provided through another buffer that exists.

4, the conversion stage (time warp module) 50 may have an associated set of control registers 53 through which suitable control parameters for the time warp conversion process may be set, e.g. , and preferably, can be set under the control of the driver 11. As shown in Figure 4, the register receives and stores, inter alia, suitable head tracking data (parameters) received from the head mounted display. Also, other control parameters representing, for example, the lenses and/or any other configuration parameters of the head mounted display may be stored in registers 53 to control the operation of translation stage (time warp module) 50 .

Of course, other methods are also possible.

FIG. 5 shows the time warp module (conversion stage) 50 of the display controller 5 shown in FIG. 4 in more detail.

As shown in Fig. 5, the time warp module (transformation stage) 50 of the display controller has multiple stages, i.e., a coordinate interpolator stage 60, a chromatic aberration correction stage 61, a "time warp" transformation stage 62 and interpolation stage 63. Each of these stages is implemented in the present embodiment in the form of a fixed-function hardware module (processing circuit) configured to perform a specific processing operation on the input data it receives, but with specific control parameters for each module. It also has some limited configurability by the ability to vary . The control parameters are, in this embodiment, given to the driver 11 for the display controller 5 when the frame sequence for the virtual reality display is to be displayed, based on the particular headset (display) 4 with which the system is being used. is set by

Using fixed function hardware modules for these stages has the advantage of being able to know their "timing" for their operation, so that the output data for display by the display controller 5 to the display 4 at an appropriate rate is Delivery can be ensured more reliably.

Of course, other methods are also possible.

Coordinate interpolator stage 60, based on the defined lens distortion corresponding to the lens distortion that will enter the output transformed surface and will be seen through the lens of the head mounted display when displayed on the display panel of display 4, time The corresponding position (coordinates) in the input frame (in the frame rendered by the GPU 3 stored in the memory 3 to be displayed to the user) for the sampling position x, y on the output transformed surface output from the warp module. Operate to determine x', y'.

The coordinate interpolator stage 60 has, for this purpose, some form of a cubic curve or elliptic function, the parameters of that function, eg the focus of the ellipse, being a driver for the display controller. It can be set using control parameters by

12 shows an example of lens distortion correction parameters for a spherical lens 130 . (This drawing has been simplified for two dimensions.)

In Fig. 12, S represents one configuration of a spherical lens as a distance along a radius from the center of the lens, and S' represents another lens configuration as a distance along a radius from the center of the lens.

In this example, S and S' both have the same lens shape, but they differ in the amount of distortion in the projection plane (represented by the dotted line shown in Fig. 12).

The parameterization shown in Figure 12 can be extended to other generalized shapes such as ellipses and cubic curves.

The chromatic aberration correction stage 61 takes the input surface positions (coordinates) x′, y′ determined by the coordinate interpolator stage 60 and “corrects” these coordinates to take into account the chromatic aberration introduced by the lens of the display 4. do".

As shown in Fig. 5, individual chromatic aberration correction is performed for each color plane (R, G, B). Also, green color plane chromatic aberration correction is used for the alpha (transparency) plane.

The output of the chromatic aberration correction stage 61 is thus three sets of corrected input surface positions (coordinates), one set being x _R ', y _R ' for use in sampling the red color plane at the input plane, and one set being A set is x _G ', y _G ' for use in sampling the green and alpha (transparency) planes from the input plane, and one set is x _B ', y _B ' for use in sampling the blue plane from the input plane. am.

Then, the corrected input surface position determined by the chromatic aberration correction stage 61 is provided to a "time warp" viewing orientation conversion stage 62.

This stage presents as its input the current head position of the user (suitably provided from the head tracking sensor of the display 4) and accordingly an input surface (frame) is rendered for display on the display 4 as if viewed therefrom. It takes as input a set of angles α, β, γ representing viewing orientations and operates to generate a projection of the input surface based on the indicated viewing orientations.

The time warp (field orientation) conversion stage 62 thus performs an additional conversion on each input surface position provided by the chromatic aberration correction stage, thereby sampling the output transformed surface (data element) that is currently being generated next. The viewing orientation transformed input surface location (x _R ", y R "), (x _G ", y _{G "} ), (x _B " , y _B ").

The transformation that the time warp (view orientation) transformation stage 62 performs on each input surface location may include, for example, an x, y shift, scaling and/or rotation. If desired, more complex transformations can be applied. In other words, it can be adjusted on-the-fly by the driver setting the control parameters for the time warp transformation stage 62, for example based on the particular headset (display) 4 the system is being used with.

The transformed input surface positions determined by the temporal warp conversion stage 62 are provided to an interpolation stage 63 that samples the input surface at the indicated positions to determine an input surface value at its respective input surface position. This is done by suitable interpolation of input surface values for defined input surface sampling (data element) locations.

Interpolation of input surface sampling locations may be performed in any suitable and desired manner. In this embodiment, bilinear interpolation is used, but other schemes may be used if desired.

As different input surface positions (coordinates) are determined for different color planes (taking into account chromatic aberrations), in this embodiment, a separate set of data is thus calculated for each color plane (the input surface determined for that color plane). Based on location), the output is interpolated with respect to transformed surface sampling locations (data elements). Thus, the RGB color data set for the output transformed surface sampling position (data element) is the determined red value from the determined "red" input surface position (coordinates) (x _R ", y _R "), and the determined "green" input surface The green value determined from the location (coordinate) (x _G ", y _G ") and the blue value determined from the determined "blue" input surface location (coordinate) (x _B ", y _B ").

These individually determined color plane (e.g., red, green, and blue) values are then used together (combined) to obtain the color for the output transformed surface sampling (data) location (x, y) in question. Provide data.

If the alpha (transparency) value is also determined, as discussed above, it is determined from the input surface location (coordinates) used to determine the green value.

The interpolated values are then sent to the output buffer (FIFO) 52 as shown in FIG. 5 for provision to the rest of the display controller's display processing pipeline for provision to the display 4 for display to the user. ) is output.

As shown in FIG. 5 , the modified input surface determined by the temporal warp transformation stage 62 (and sampled by the interpolation stage 63) such that suitable input surface data is available to the interpolation stage 63. It is used to control the fetching of the appropriate region of the input surface containing this sampling position into the input buffer 51 before the position is used by the interpolation stage 63 (65).

To do so, the integer part of the input surface position determined by the temporal warp transformation stage 62 is used to identify the appropriate two-dimensional block of the input surface containing the required input surface data. The input stage then operates to load each of these input surface blocks into the input buffer 51 . This is further illustrated in FIG. 7 discussed below.

As also shown in FIG. 5, interpolation stage 63 requests input surface data into input buffer 51 for use by interpolation stage 63 and interpolation stage 63 interpolates that data to provide output transformed surface sampling locations. There may be a suitable delay (delay absorption) mechanism 64 included in the conversion stage to allow for a delay between processing . This can be implemented as desired.

As shown in Fig. 5, receipt of a suitable synchronization signal 65 from the display 4 triggers the operation of the time warp module (transformation stage), the timing of which is controlled.

FIG. 6 is a flowchart showing the operation of the time warp module (conversion stage) 50 as shown in FIG.

Figure 6 shows the operation for each pixel (data) position in the output converted surface to be created for display. This operation is repeated for each required output transformed surface pixel (data) location.

As shown in Figure 6, the process begins with an appropriate sync signal 120 indicating the start of a new output frame for display.

Next, the desired output transformed surface pixel coordinates x ₀ , y ₀ to be generated next are determined (step 121).

Then, the coordinate interpolator stage 60 outputs the transformed frame when displayed on the display panel of the display 4, based on the defined lens distortion corresponding to the lens distortion that will be seen through the lens of the head mounted display. Output to be output from the time warp module In the input frame for pixel (sampling) position x ₀ , y ₀ on the transformed surface (in the frame rendered by GPU 2 stored in memory 3 to be displayed to the user) ) to determine the corresponding positions (coordinates) x _i , y _i (step 122).

Then, the chromatic aberration correction stage 61 takes the input surface position (coordinates) x _i , y _i determined by the coordinate interpolator stage 60 in step 122 and considers the chromatic aberration introduced by the lens of the display 4. "corrects" these coordinates so that 3 sets of corrected input surface positions (coordinates) x _r , y _r ; x _ga , y _ga ; Gives x _b , y _b .

Then, in step 123, the corrected input surface position determined by chromatic aberration correction stage 61 is "time warped" by translation stage 62 based on a set of angles α, β, γ representing the user's current head position. Takes coordinate reprojection (calibration), thereby providing a set of orientation-transformed input surface locations (x' _r , y' _r ), (x' _ga , y' _ga ), (x' _b , y' _b ) do.

In this embodiment, input head positions representing angles α, β, γ are updated based on head tracking information provided from the display for each new line of output time warp transformed surface to be created. This allows the time warp conversion process to be adjusted on a line-by-line basis, instead of having to wait until the start of the next output frame to be displayed, for example. Of course, other methods are also possible.

Then, the input surface position thus determined is the input surface position containing the necessary input surface sampling position data so that data values for the determined input surface position can be determined by the interpolation stage 63, as shown in FIG. 6 . Used to request a fetch of the appropriate block into the input buffer. It uses the integer part of each determined input surface location to identify which input surface block will be needed. The identified input surface blocks are then fetched and stored in the input buffer (step 126).

The interpolation stage then reads the appropriate input surface data location from the input surface block stored in the input buffer and interpolates the appropriate data value for the output surface pixel (sampling location) at (x ₀ , y ₀ ) in question. (step 127).

The output pixels thus generated are then collected in an output buffer for subsequent presentation to the display compositing unit (and use by the display compositing unit) and presentation to the display for display (step 128).

5 and 6, the time warp module (conversion stage) of the display controller 5 (on a sampling position-by-pixel (pixel-by-pixel) basis) outputs a sequence of sampling positions (pixels) on the transformed surface to the user. Creates an output transformed surface to be displayed.

In this embodiment, the time warp module (conversion stage) is configured to generate the output transformed surface raster line by raster line (the order in which it will need to present the output transformed surface to the display 4 for display). because it wins).

Figure 7 illustrates this and shows three raster lines 71 of an exemplary output transformed surface to be provided as output streams from interpolation stage 63 of the time warp module (transformation stage).

7 also shows the 16×16 sampling positions (pixels) that are individually loaded into the input buffer for use by the interpolation stage 63 while the lines 71 of the output transformed surface are being generated, as discussed above. ) shows the corresponding input surfaces 72 stored in suitable frame buffers (eye buffers) in the memory 33, divided into an array of blocks 73.

As shown in Fig. 7, due to distortion, typically barrel distortion, introduced by a lens in the head mounted display 4 through which the viewed displayed output transformed surface passes, each horizontal line at the transformed output surface The raster line 71 will in fact correspond to the curve on the input surface 72 . Thus, as shown in FIG. 7 , raster lines 75 on the output transformed surface will correspond to curves 76 on the input surface 72 and so on.

Thus, as shown in FIG. 7 , to generate raster lines 75 at the output transformed surface, a line across the input surface 72 (rather than simply a linear row of blocks from the input surface 72) A number of different blocks 73 will need to be loaded into the input buffer. This is illustrated on the right in figure 7 showing the contents 77 of the input buffer needed to process the raster lines 75 in the output transformed surface.

Figure 7 also shows the contents 78 of the input buffer for blocks from the input surface 72 required when generating raster lines 79 on the output converted surface, and the input when generating raster lines 81. Shows the contents 80 of the buffer.

As discussed above, the appropriate blocks from the input surface 72 are sent to the input buffer in response to signals from the time warp conversion stage 62 indicating the input surface positions that will be needed to generate each sampling position in the output transformed surface. Loaded. The loading of blocks into the input buffer is controlled in such a way that any blocks needed for the next line on the output converted surface are held in the input buffer until they are no longer needed for the next line on the output converted surface.

As can be appreciated from the above, the present invention, at least in its preferred embodiment, will provide a more efficient mechanism for providing a user using a head mounted virtual reality display system with a suitable "time warped" image for display. can

This is accomplished, at least in the preferred embodiment of the present invention, by performing a "time warp" of frames to be displayed at the display controller as part of the display controller display processing pipeline operations. In particular, in at least a preferred embodiment of the present invention, the display controller is capable of providing the desired "time warped" frame for display as a pixel stream directly to the display for display, on a sampling position-by-pixel (pixel-by-pixel) basis. It is configured to generate a time warp transformed surface for display from the surface.

The foregoing detailed description has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the subject technology to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The described embodiments were chosen to best illustrate the principles of the present technology and its practical application, thereby allowing others skilled in the art to make various modifications in the various embodiments and tailored to the particular use contemplated. You can make the best use of this technology while you have it. It is intended that the scope be defined by the claims appended hereto.

Claims (25)

A display controller for a data processing system comprising:
a local input buffer operable to store data of the input surface;
an input stage operable to fetch data of an input surface from memory into the local input buffer;
an output stage operable to provide an output surface for display on a display; and
identify data regions of the input surface for which data is needed to provide an output transformed version of the input surface, cause the input stage to fetch the identified data regions of the input surface from the memory into the local input buffer, and processing input surface data from the local input buffer to transform data of the input surface based on received view orientation data to provide the output transformed version of the input surface as an output transformed surface; , a conversion stage operable to provide the output transformed surface to the output stage for presentation as an output surface for display to a display.
According to claim 1,
wherein the input stage is operable to fetch block data of the input surface from the memory into the local input buffer.
According to claim 1 or 2,
wherein the conversion stage is configured to periodically update its conversion operation based on newly received viewing orientation data.
According to claim 1 or 2,
wherein the transformation stage is operable to transform the input surface based on a distortion to be caused by a lens or lenses through which the displayed output transformed surface to be viewed by a user passes.
According to claim 1 or 2,
wherein the transformation stage is operable to transform the input surface based on chromatic aberrations to be caused by a lens or lenses through which the displayed output transformed surface to be viewed by a user passes.
According to claim 5,
and wherein individual chromatic aberration corrections are determined for different color planes of the input surface.
According to claim 1 or 2,
wherein the conversion stage is operative to determine, for a data location on the output transformed surface, a corresponding location on the input surface from which data for that data location on the output transformed surface should be taken.
According to claim 1 or 2,
The conversion stage,
a coordinate interpolator stage operable to determine, relative to a data location on the output transformed surface to be created, a corresponding data location on the input surface based on the defined lens distortion;
a chromatic aberration correction stage operable to correct the input surface location determined by the coordinate interpolator stage to account for chromatic aberration when viewing the output transformed surface to provide at least one corrected input surface location; and
The at least one corrected input surface determined by the chromatic aberration correction stage based on received viewing orientation data to provide an output input surface position that is sampled to provide data for use with respect to the data position of the output transformed surface. A display controller comprising a viewing orientation translation stage operable to further correct position.
According to claim 1 or 2,
wherein the conversion stage further comprises an interpolation stage operable to interpolate a plurality of input surface sampling location values to provide interpolated sampling location values for use with data locations on the output transformed surface.
According to claim 1 or 2,
wherein the conversion stage is operative to produce output transformed surface data as a stream of output transformed data locations.
According to claim 1 or 2,
wherein the conversion stage includes fixed-function processing circuitry, wherein the fixed-function processing circuitry is configurable by setting one or more control parameters for the circuitry during use.
A virtual reality display device comprising the display controller according to claim 1 or 2.
An apparatus for transforming an input surface to provide an output transformed surface based on received view orientation data, comprising:
a local input buffer operable to store data of the input surface;
a coordinate interpolator stage operable to determine, for a data position on an output transformed surface to be output from the device, a corresponding data position on an input surface based on a defined lens distortion;
a chromatic aberration correction stage operable to correct the input surface location determined by the coordinate interpolator stage to account for chromatic aberration when viewing the output transformed surface to provide at least one corrected input surface location; and
To the chromatic aberration correction stage based on received viewing orientation data to provide an output input surface position that is sampled from input surface data from the local input buffer to provide data used for the data position of the output transformed surface. further modify the at least one modified input surface location determined by, identify a data region of the input surface for which data is needed to provide the output transformed surface, and store the identified data region of the input surface in memory. and a view orientation conversion stage operable to fetch from the local input buffer.
A method for operating a display controller in a display processing system,
The display controller comprises a local input buffer operable to store data of an input surface, an input stage operable to fetch data of an input surface from memory into the local input buffer, and an input based on received view orientation data. a transformation stage operable to transform a surface to provide a transformed version of the input surface as an output transformed surface;
The method,
The conversion stage of the display controller,
identifying data regions of the input surface for which data is needed to provide an output transformed version of the input surface, fetching the identified data regions of the input surface from memory into the local input buffer, and processing input surface data from the local input buffer to transform data of the input surface based on received viewing orientation data to provide the output transformed version as an output transformed surface; and
A method of operating a display controller in a display processing system comprising the step of the display controller providing the output converted surface to a display for display.
According to claim 14,
wherein the input stage fetches a block of data on the input surface from the memory into the local input buffer.
The method of claim 14 or 15,
The method of operating a display controller in a display processing system further comprising the step of the transform stage periodically updating its transform operation based on newly received viewing orientation data.
The method of claim 14 or 15,
operating a display controller in a display processing system, wherein the transformation stage further comprises transforming the input surface based on a distortion to be caused by a lens or lenses through which the displayed output transformed surface to be viewed by a user passes. method.
The method of claim 14 or 15,
operating a display controller in a display processing system, wherein the transformation stage further comprises transforming the input surface based on chromatic aberrations to be caused by a lens or lenses through which the displayed output transformed surface to be viewed by a user passes. method.
According to claim 18,
A method of operating a display controller in a display processing system in which individual chromatic aberration corrections are determined for different color planes of the input surface.
The method of claim 14 or 15,
The conversion stage is configured in a display processing system to determine, for a data location on the output transformed surface, a corresponding location on the input surface from which data for the data location on the output transformed surface should be taken. How to operate the controller.
The method of claim 14 or 15,
The conversion stage,
for a data position on an output transformed surface to be output, determine a corresponding data position on the input surface based on a defined lens distortion;
when viewing the output transformed surface to provide at least one corrected input surface position, modifying the determined input surface position to account for chromatic aberration; and,
The at least one modified input surface location determined, based on received viewing orientation data, to provide an output location on the input surface that is sampled to provide data used for the data location of the output transformed surface. By further modifying
A method for operating a display controller in a display processing system that transforms the input surface to provide the output transformed surface based on received viewing orientation data.
The method of claim 14 or 15,
operating a display controller in a display processing system further comprising the conversion stage interpolating a plurality of input surface sampling position values to provide interpolated sampling position values for use with data positions on the output transformed surface. how to do it.
The method of claim 14 or 15,
wherein the conversion stage produces the output transformed surface data as a stream of output transformed data locations.
The method of claim 14 or 15,
configuring, by a driver for the display controller, one or more control parameters for the conversion stage to configure the stage in use.
A computer program, stored on a computer readable storage medium, comprising computer software code for performing a method according to claim 14 or 15 when executed on a data processor.

Images