TW202023272A - Apparatus, systems, and methods for foveated display - Google Patents

Abstract

Systems and methods of foveated light modulation may take advantage of the eye’s exponential reduction in acuity/resolution as it moves outward from the center of gaze by providing a plurality of display zones (Zone 0, 1, 2, … N). Particularly, the system may include a processor that couples to receive image data relating to an image and foveation zones to generate a foveated image frame having header packet data, which identifies two or more zones of differing resolution. Each zone may be defined by a plurality of macropixels, having a corresponding macropixel ratio. The system may further include a driver controller circuit that couples to receive the foveated image frame to generate modulation planes. One or more modulation devices may couple to receive the modulation planes to generate an expanded dataset based upon the corresponding macropixel ratio, such that the foveated image is output to a display.

Description

Gaze point display device, system and method

The invention relates to a fixation point display device, system and method.

Software imaging applications using eye tracking can concentrate the pixel density of the display device in the area of interest, where the center area is consistent with the user's gaze direction, and blocks are generated around the area of interest with a lower resolution , To surround the area around the eye. This is the so-called focal point imaging (foveated imaging) or focal point rendering (foveated rendering). Even if you use Head Mounted Display (HMD) applications without eye tracking, you can use gaze point imaging to concentrate the pixel density in the center of the display and use a lower resolution to fill the field of view in the display (Field of View, FOV). By reducing the surrounding image details and/or resolution, gaze point imaging can reduce the number of pixels that need to be processed and transmitted, which can make the processing speed of the host faster, and the display device can perform imaging faster under ideal conditions .

However, current display devices generally require a video source with stable resolution, in which each row needs to have the same number of pixels in each row. In addition, in order to match the function of the display device, the host must convert the gaze point image into a stable resolution image, and then use the standard video protocol to transmit this image to the display device. This means that the surrounding low-resolution blocks will be scaled up or copied to match the central high-resolution blocks. Therefore, although gaze point imaging reduces the processing requirements of the host, it does not achieve the corresponding savings on the display device. Further, the bandwidth required to transmit the image to the display device is still very high, and the time for writing the pixel array in the display device remains the same in real value. Some systems reduce the image transmission bandwidth by allowing the host to add compression hardware/algorithms, and the display needs to add decompression functions. Although these compression methods can be used to reduce the data bandwidth between the host and the display driver, these compression methods do not reduce the data transmission bandwidth between the driver and the display device or reduce the display device during the process of writing data to the pixel array. The processing time between internal components in the middle, which may limit the maximum frame rate, maximum bit depth, and/or maximum array size.

Some systems try to imitate the form of fixation imaging by dividing the video into two channels leading to different display devices: one channel is used for the high-resolution area of interest, and the other channel is used for the low-resolution periphery/ background. These systems accomplish this by mechanically manipulating the projection optics in the direction of the gaze point for high-resolution areas of interest.

Existing display devices that do not have a fixation point function generally accept a single uniform resolution to cover the entire display. It is usually a one-to-one match of the physical layout of the display device (X horizontal pixels by Y vertical pixels). Even if the display can be configured to run in data scaling or replication mode to fill the display with a lower input resolution, the input resolution remains constant in the image (for example, a The input pattern is mapped with a quarter of the original block resolution, where each input pixel is represented as four squares of the same color display pixels). Therefore, if an existing system with a fixation host is connected to a display without fixation function, the host must be configured to transmit the highest resolution matching this display; copy the low-resolution block to match the display resolution; and then Transmit high-resolution data for the entire image/display.

Due to various factors, the disadvantage of the existing system of fixation point imaging is the lack of efficiency and effectiveness. First, the existing fixation point systems and methods include redundant data, which wastes bandwidth, limits the frame update rate, and limits the original bit depth and/or maximum pixels of each display. Second, since the block size and offset parameters are not included in the video data, when the gaze point changes, the gaze point image cannot be updated in real time frame by frame using the static display resolution/configuration. Third, the standard video protocol does not define the mixed resolution frame or handshake foveation parameters of the display hardware function. Finally, there is no support for multi-line writing of the same data. In some cases, multi-line writing can cause timing errors and/or local overloads, resulting in more than two to one copy volumes.

A number of embodiments are presented below.

An embodiment of an apparatus, system, and method for gaze point display is provided. It should be understood that this embodiment can be implemented in a variety of ways, such as a program, device, system, device, or method. Several invention embodiments will be described below.

In some embodiments, a gaze point display system is provided. Gaze point rendering is a type of image processing or image generation that uses the acuity of the eyes by providing multiple display areas (such as area Z0, area Z1, area Z2, area Z3, etc.) The resolution decreases exponentially when moving from the center of the retina (the direction of the user's gaze) to the periphery of the image. The gaze point display system described herein provides a unique method and protocol for encapsulation of gaze point image frames, and an innovative method for processing gaze point writing to display gaze point images on a display device. In particular, the system may include a processor coupled to receive input image data and gaze area definition data to generate a rendered gaze point image, and then process the rendered gaze point image using the selected protocol according to the present invention It becomes a gaze point image frame with image data and header packet data, which indicates two or more blocks with different resolutions. Each block can be defined by macro pixels with a corresponding macro pixel ratio. For example, in a gaze point image with three blocks, the first block Z0 may have a one-to-one horizontal to vertical ratio, and the second and third blocks (Z1 and Z2) may have corresponding One to two and one to four megapixel ratios (where the Z1 megapixel is a 2x2 display pixel array, and the Z2 megapixel is a 4x4 display pixel array). The system may further include a drive controller circuit coupled to receive the gaze point image frame to generate gaze point bit-plane data, and convert the bit-plane data into multiple modulation planes. One or more modulation devices can be coupled to receive multiple modulation planes to generate an expanded data set, so that the gaze point image can be generated on a display or output to a display device. In the expansion process of the modulation plane data, for the data of the block with reduced resolution, based on the corresponding macro pixel ratio related to the block, a single bit of multiple related macro pixels can be copied to a group Display pixels.

In some embodiments, a method and protocol for gaze point display are provided. The method may include receiving image data about the image and the gaze area. For example, based on real-time user's retina and/or head position, a processor can receive image input data and tracking data. In response, the method may further include generating a gaze point image frame based on the image data, wherein the gaze point image frame includes header packet data that indicates two or more concentric blocks of different resolutions, Thus, each block is compressed to be defined by the ratio of multiple megapixels to one corresponding megapixel. The method may further include transmitting the gaze point image frame to one or more modulation devices having raster logic, which is coupled to a display circuit including a pixel array. For example, the gaze point image frame with the header packet data can be transmitted to the driver controller circuit, which generates the gaze point bit plane data and transfers the gaze point data based on the modulation format and the header packet data. The viewpoint bit plane is converted into multiple modulation planes. The modulation planes are transmitted to the one or more modulation devices. Then, the method may include using raster logic based on the ratio of the header packet data to the corresponding giant pixel to write the modulated plane data into the display pixel array, where for the gaze area with reduced resolution, based on the correlation with The corresponding macro-pixel ratio of each block, a single bit of the related multiple macro-pixels is copied to a subset of the display pixel array.

In some embodiments, a tangible, non-transitory computer-readable medium has a plurality of instructions, and when the processor executes the instructions, the method for displaying the gaze point is executed. The gaze point display method may include receiving image data about the image and multiple gaze point regions. For example, a processor can receive image input data and tracking data, based on real-time user's retina and/or head position, to generate a rendered gaze point image, just like the gaze point rendering done in the industry method. In response, the method may further include generating a gaze point image frame based on the rendered gaze point image data and the selected transmission protocol, wherein the gaze point image frame includes header packet data that indicates two or more Concentric blocks with different resolutions, so that each block is defined by multiple megapixels and the corresponding megapixel ratio. The method may further include transmitting the gaze point image frame to one or more modulation devices, and decoding the frame into an extended data set for controlling a display circuit with a pixel array. For example, the gaze point image frame with header packet data can be used to generate gaze point bit-plane data, which is converted into multiple modulation planes based on the modulation format and the header packet data. These modulation planes can be transmitted to the one or more modulation devices and decoded to this extended data set. Next, the method may include using raster logic based on the expanded data set through the ratio of the header packet data to each corresponding macro pixel to write the modulated plane data into the display pixel array; In the fixation area, based on the corresponding macro-pixel ratio associated with each block, a single bit of the related multiple macro-pixels is copied to a subset of the display pixel array.

In some embodiments, a gaze point display device is provided. The gaze point display device may include a drive controller circuit coupled to receive a gaze point image frame to generate a plurality of modulation planes, wherein the gaze point image frame includes header packet data, and the header packet data indicates two One or more blocks with different resolutions, and each block is defined by a plurality of macro pixels and a corresponding plurality of macro pixel ratios. Further, the gaze point display device may include one or more modulation devices coupled to receive the modulation planes, so that a gaze point image is generated on a display, wherein for a plurality of gaze points with reduced resolution For a block, based on the corresponding macro pixel ratios related to the block, a single bit of the related multiple macro pixels is copied to a group of display pixels.

Other aspects and advantages of the embodiments will become apparent through the following detailed description in conjunction with the accompanying drawings, which illustrate the principles of the described embodiments by way of example.

The following embodiments describe the device, system and method for gaze point display. Those skilled in the art should understand that these embodiments can be implemented without some or all of these specific details. In other cases, the conventional processing operations are not described in detail to avoid unnecessary confusion in the embodiments.

In some embodiments, a gaze point display system is provided. Foveated rendering is a type of image processing and image generation, which uses the exponential reduction of the acuity/resolution of the eye moving from the center of the retina (the user's gaze direction) to the periphery of the image To provide multiple display zones [such as zone 0 (Z0), zone 1 (Z1), zone 2 (Z2), zone 3 (Z3), etc.]. The gaze point display system described herein provides a method system, a gaze point image frame packaging protocol, and an innovative method for processing gaze point writing for displaying the gaze point image on a display device. This fixation point writing is unique to this fixation point display system and method, and solves the final bandwidth bottleneck. In particular, the system includes a processor, which is coupled to receive input image data and gaze area definition data to generate a rendered gaze point image, but the gaze point image is processed to have The gaze point image frame of the image data and the header packet data identifies two or more blocks of different resolutions. Each block can be defined by multiple macro pixels with corresponding macro pixel ratios. For example, in a gaze point image with three blocks, the first block Z0 may have a horizontal and vertical ratio of 1:1, and the second and third blocks (Z1 and Z2) may have corresponding macros. The pixel ratio is 1:2 and 1:4 (where the Z1 giant pixel is a 2x2 display pixel array, and the Z2 giant pixel is a 4x4 display pixel array). The system further includes a drive controller circuit coupled to receive the gaze point image frame to generate gaze point bit-plane data, and convert the bit-plane data into a modulation plane. One or more modulation devices (such as a display or liquid crystal-on-silicon (LCoS) display) can be coupled to receive the modulation plane, thereby generating an expanded data set, which can be written into the pixel array , Thereby generating a fixation point image on the display. In the expansion of the modulation plane data for a block with a reduced resolution, based on the corresponding macro pixel ratio associated with this block, a single bit of related multiple macro pixels can be copied to a group of display pixels.

In some embodiments, a method and protocol for gaze point display are provided. The method may include receiving image data associated with the image and the gaze area. For example, the processor can receive image input data and tracking data in real time based on the user's retina and/or head position. In response, the method may further include generating a gaze point image frame based on the image data, wherein the gaze point image frame includes header packet data indicating two or more concentric blocks with different resolutions, thereby compressing each block (Zone) is defined by multiple megapixels and the corresponding megapixel ratio. The method may further include transmitting the gaze point image frame to one or more modulation devices, the one or more modulation devices having a raster logic coupled to a display circuit including a pixel array. For example, the gaze point image frame with header packet data can be transmitted to the driver controller circuit, which generates the gaze point bit plane data, and is based on a modulation scheme and header packet Data and convert the gaze point bit plane data into a modulation plane. The modulation planes can be transmitted to the one or more modulation devices. In addition, the method may further include outputting the gaze point image to the display pixel array using raster logic based on the ratio of the header packet data to each corresponding giant pixel, where the gaze point area with reduced resolution is based on the correlation The corresponding macro-pixel ratio of each block copies a single bit of multiple related macro-pixels to a subset of the display pixel array.

The fixation point display system and method provide two protocol methods for fixation point transmission: zone order (Zone-Order) and raster order (Raster-Order). Each method can have multiple formats to define specific protocol standards (described in detail below). This fixation point display system and method use two protocol stages: image frame interface (each pixel data between the host and the display driver; for example, 24 bits/pixel) and the modulation plane interface (discrete between the driver and the display Data; for example, 1 bit/pixel). The gaze point display system disclosed here applies these protocols to two applications: center-detail and peripheral detail. Further, this gaze point display system provides multiple gaze point writing implementations: single column, double column, four column, etc.

Advantageously, this gaze point display system and method propose to change the display device to accept the innovative gaze point protocol; therefore, it can save the transmission bandwidth and save the writing time of the pixel array of the display device. In addition, the gaze point display system and method described herein can have a higher video rate, a higher bit depth, and/or a higher display resolution. Further, the gaze point display system and method proposes some discrete constraints for gaze point processing to match or simplify the implementation in the display device. In addition, the gaze point display system and method keep the shape of the block in a rectangular block to match the physical array structure of the display device pixels, thereby improving the existing system (due to the optical characteristics, projection on the viewing zone can make these blocks become Non-rectangular/non-linear). According to the relative size of each block, the gaze point image processing method of the present invention described herein can generally reduce the transmitted image or image frame data by one-sixth to one-tenth of the total display pixels. For example, in the gaze point image frame, a 2K×2K display with four megapixels may only need to input 0.5 megapixels of giant pixels.

In the following description, many details are explained. However, it is obvious to a person with ordinary knowledge in the art that the present invention can be implemented without these specific details. In some cases, to avoid blurring the focus of the present invention, only block diagrams rather than detailed diagrams are used to represent conventional structures and devices.

Some parts of the detailed description below are presented based on algorithms and symbolic representations of data bit operations in computer memory. The description and representation of these algorithms are the means used by those skilled in the data processing field to most effectively convey the essence of their work to others in the field. Here, an algorithm is generally considered to be a self-consistent sequence step to obtain the desired result. These steps are those requiring physical manipulation of physical quantities. Although not necessary, these physical quantities usually take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. Mainly for general purpose, it is convenient to refer to these signals as bits, values, elements, symbols, characters, vocabulary, numbers, etc.

However, it should be borne in mind that all these and similar terms should be associated with the appropriate physical quantities and are merely convenient labels applied to these physical quantities. Unless explicitly stated otherwise, it is obvious from the following discussion that in the entire specification, use such as "retrieved", "generated", "detected", "written", "converted", "received", "extended" "Analysis", etc. refers to the operation and process of a computer system or similar electronic computing equipment, which converts data expressed as computer system registers and physical (electronic) quantities in the memory into computer system memory or registers Or other data represented by physical quantities in other such information storage, transmission or display equipment.

The present invention relates more to a device for performing the operation herein. The device may be specially constructed for the required purpose, or it may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such computer programs can be stored in computer-readable storage media, such as but not limited to any type of magnetic disk, including floppy disks, optical disks, CD-ROM and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical card or any type of medium suitable for storing electronic instructions, each of which is coupled to the computer system bus.

Mentioning "one embodiment" or "an embodiment" in the specification means that a specific feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present invention. The words "in one embodiment" in different positions in this specification do not necessarily refer to the same embodiment. Throughout the description of the drawings, the same reference numerals represent the same elements.

1A, according to some embodiments, a gaze point electromagnetic radiation modulation system with gaze point light modulation is disclosed. Gaze point rendering is a type of image processing or image generation, which uses the acuity/resolution of the eyes from the center of the retina (the user’s gaze direction) by providing multiple display areas (such as areas Z0~Z3) Moving outward to the periphery of the image decreases exponentially. In the embodiment of the present invention, the block may be rectangular. However, those with ordinary knowledge in the field can understand that the shape of the block can be changed. The gaze point display system described herein provides a method and protocol for encapsulating gaze point image frames, and a processing method for writing gaze points for displaying the gaze point image on a display device. In particular, the gaze point electromagnetic radiation modulation system 100 (such as the gaze point light modulation system) may include a processor (central processing circuit and/or graphics processing circuit) 110, a drive controller circuit 120, and one or more modulations.装置130 and display 146. The processor circuit 110 has a block definition module 111, a gaze point rendering module 112, a gaze point image memory 114, and an image protocol encoding logic 115, which are generally used to receive input from one or more input data sources 105 To generate the gaze point image frame 103, the gaze point image frame 103 has header packet data, which defines two or more concentric blocks with different resolutions, where each block is composed of multiple macro pixels and corresponding Defined by the megapixel ratio; one ratio is used in the horizontal direction and the other ratio is used in the vertical direction. In an embodiment of the present invention, the ratio of each macro pixel is an integer. In some embodiments, the input from the one or more input data sources 105 includes one or more images and related gaze block data. The "gazing point image" described here generally refers to an image or video frame, which is divided into two or more resolution blocks. In some embodiments, the processor 110 may be included in a host computer (not shown), so that the host computer transmits the gaze point image frame 103 to the drive controller circuit 120 associated with the display 146. The driving controller 120 is generally used to control the modulation device circuit 130 so that the gaze point image is output to the display 146 based on the input data 105 (for example, image data). Of course, it should be understood that the drive controller circuit 120 and/or the processor circuit 110 may also include other conventional and/or dedicated circuits and/or logical structures, such as frame buffer memory/cache, sequential circuits, vertical /Horizontal scan line circuit, processor circuit, etc. The gaze point image (such as the projected gaze point image or the direct gaze point image) stored in the memory 114 is output to the display and/or generated or rendered on the display 146. The gaze point image may include multiple resolutions Blocks Z0, Z1, Z2, Z3, ... ZN, each of which has a different resolution. For example, according to the present invention, the center detail mode (center detail mode) can be used to generate these blocks, where the block Z0 with the highest resolution tracks the user's gaze direction, and the other blocks (Z1, Z2, Z3,… ZN) has a lower resolution than the block Z0, so that the resolution of each block decreases sequentially from the user's fixed point. In other words, the resolution of the second block Z1 is lower than the resolution of the block Z0; the resolution of the third block Z2 is lower than the resolution of the block Z1; the resolution of the fourth block Z3 is lower than the resolution of the block Z0 The resolution of Z2, and so on. As those with ordinary knowledge in the field can understand, the description of the aforementioned four blocks is only for illustration, and the number of blocks and/or the size of blocks relative to adjacent blocks can be changed. The teaching of the present invention can be equivalent to a system with N fixation point image blocks. According to the gaze point display system and method, the display 146 may be an amplitude and/or phase display. The application of the foveated light modulation system 100 disclosed in the present invention may generally include, for example, target applications, such as holographic photography for heads up displays (HUDs), and head-mounted displays (head-mounted displays). displays, HMDs) augmented reality (AR), mixed reality (MR) or virtual reality (VR), etc. Of course, these applications are provided for illustration only, and not for limiting the present invention. In the embodiment of the present invention, the data in a giant pixel is not different from the data in a pixel. In an embodiment of the present invention, when the gaze point rendering module 112 creates pixels and puts them into the memory, they may be, for example, 24 bits (for example, full color bits). Multiple macro pixels correspond to multiple display pixels, and the image rendering process establishes pixels, which may also be called macro pixels. The encoding module 115 does not generate giant pixels, but rearranges the pixels into an order indicated by a selected protocol or a specific protocol. The modulation plane pixel is a single bit or corresponds to a single bit.

In the embodiment of the present invention, the block may be rectangular, which is related to the row-column structure of the pixel array of the display. However, those with ordinary knowledge in the art should understand that the shape of the block is changeable according to the structure and characteristics of the gaze point display, so as to facilitate the reproduction of data according to the macro pixel ratio. In addition, in one embodiment of the present invention, the ratio of macro pixels in the horizontal and vertical dimensions are both integers, which allows the macro pixel data to be copied to the entire and/or individual display pixels. Those with ordinary knowledge in the art should understand that when megapixel values are applied or written to multiple or other display pixels, the grayscale device may allow for scaling pixel data or filtering/processing pixel values in other ways.

The gaze point image frame 103 generated by the processor circuit 110 usually includes a gaze point area, and can be packaged using conventional and/or dedicated image transmission protocols (for example, the display string from the Mobile Industry Processor Interface (MIPI) Alliance). Line interface (Display Serial Interface, DSI), high-definition multimedia interface (HDMI), display interface (DisplayPort), etc.). In order to communicate with a display, a display device with a fixation point display or fixation point image function, the processor circuit 110 may embed header and/or command information (such as mode, action and/or format selection information) into the fixation point image Frame 103. The header information may include, for example, the number of gaze area in use, the resolution of each block, the size and position of the gaze area, the data sequence, the packaging format, the expected display capability, etc. The size and location of each block can be selected by the processor circuit 110, and can be selected based on, for example, the array size and other attributes of the modulation device circuit 130, the optical characteristics of the system, the tracking logic 107, the rendering algorithm, the operating environment, and the like. The processor circuit 110 may generate the gaze point image frame 103 based on one or more input data sources 105. The image data source may include, for example, an image sensor (such as a camera) to capture environmental image data, image overlay data, and so on. The input data source 105 may include, for example, multiple image sensors to capture image data with different resolutions. For example, the gaze point image frame 103 may include three blocks: a first block Z0, a second block Z1, and a third block Z2. In some embodiments, for example, the first zone Z0 may have the highest resolution (for example, the one-to-one correspondence between the gaze point image and the pixel), and the second zone Z2 may have a lower resolution than the first zone. Resolution, such as a four-to-one pixel resolution, which has a quarter of the resolution of the first block (one-half in the horizontal and one-half in the vertical), and the third block Z3 can have A lower resolution than the second block, such as a sixteen to one resolution, which has one sixteenth of the resolution of the first block (a quarter is horizontal and a quarter is vertical). These blocks can be defined as binary multiples of the highest resolution blocks (two to one, four to one, sixteen to one, etc.). In some embodiments, the processor circuit 110 may refresh the blocks of the coded gaze point image frame 103 for each frame, each subframe, and/or on a predetermined basis such as one frame interval.

In some embodiments, the processor 110 may be coupled to receive user retina and/or head tracking data from the tracking logic module 107. The tracking logic module 107 has real-time or stored tracking software motors, electronics, and /Or mechanical components. In particular, the block definition module 111 can use the system optical parameters and data from the tracking logic 107 together. The tracking logic 107 can sense the user's retinal position and generate gaze zone parameter data corresponding to the sensed retinal position . According to the specific embodiment, the gaze point rendering module 112 can be coupled to the tracking logic 107 to receive the user's gaze point based on retinal gaze; wherein the gaze point rendering module 112 can use the gaze point rendering algorithm based on one or more parameters Method to calculate the size and position of each block. The one or more parameters are: total field-of-view, optical system distortion, fovea acuity, Tolerance of tracking logic (tolerance of tracking logic), latency of tracking logic (latency of tracking logic), rate of motion, etc. In some embodiments, the tracking logic 107 may be included in the system 100 to define the position of each block in the image frame, wherein the tracking logic 107 is used to track and locate the position of the eyes and/or the head. Those with ordinary knowledge in the art should understand that any logic of the present invention can be implemented by motors, electronic and/or mechanical components.

In some embodiments, the driving controller circuit 120 may include a memory 122, a first conversion unit 124 and a second conversion unit 126. The first conversion unit 124 may be coupled to receive the gaze point image frame, and in response to generate gaze point bit plane data based on the gaze point image frame. The second conversion unit 126 may be coupled to receive the gaze point bit-plane data, and in response, generate a modulation plane 127 based on the gaze point bit-plane data and an associated modulation scheme (associated modulation scheme). In some embodiments, the memory 122 can store the gaze point image frame, the gaze point bit plane data, or the modulation plane 127. Each of the one or more modulation devices 130 has a display pixel array 144 and can be coupled to receive the modulation plane 127 and output a gaze point image on the display 146. In particular, the one or more modulation devices 130 may expand each line-set of the modulation plane based on the header packet data; among them, for blocks with reduced resolution, based on Regarding the block-related macro pixel ratio, a single bit of multiple related macro pixels is copied to a subset of the display pixel array.

In some embodiments, the modulation device circuit 130 may include, for example, a protocol decoding logic 133, a display circuit 140 (such as an LCOS display device, a panel, a display panel, or a spatial light modulator), a raster logic 150, and a memory 132. In some embodiments, the decoding logic 133 is coupled to receive the modulation plane from the drive controller circuit 120 to parse the header packet data, and the raster logic 150 generates an extended data set based on the control from the decoding logic 133, which may include The header packet data and the corresponding megapixel ratio. The raster logic 150 may include a row buffer 156 for holding each row of each line group during the decoding phase of the operation; and a row-to-column 158 for holding the expanded data set to be written to the pixel array 144, so that the display 146 display effect. The raster logic 150 may further include a line group collection circuit 152 or direct write logic 154. The display circuit 140 may be coupled to receive the expanded data set, and in response, display the gaze point image on the display 146. In particular, the display circuit 140 may include a control unit coupled to receive the expanded data set and generate a plurality of corresponding binary values to be applied to the pixel array circuit 144, wherein the corresponding binary values control the propagation through each pixel The amplitude and phase of electromagnetic radiation. In some embodiments, the display circuit 140 may include, for example, a liquid crystal on silicon (LCoS) display circuit (not shown), such as a display provided by Compound Photonics. Depending on the conditions required for a given application, the display circuit 140 may include a phase type and/or an amplitude type.

In FIGS. 3A and 3B, the line group collection circuit 152 will be described in further detail as a part of the memory that is read from the buffered data or immediately from the received raster sequence data. In some embodiments, the display device may provide a direct write logic 154 to be able to write a part of a row corresponding to a block without affecting other parts of the row. This function allows to write each block directly in the order of the blocks without having to collect giant pixels from different blocks in the line group. However, it uses multiple row write times to write all parts of the row, which may be the limiting factor of this embodiment.

In some embodiments, the header packet data may include a resolution-order toggle bit to enable the center detail mode and the peripheral detail mode. During the operation in which the center detail mode is activated, the gaze point image frame includes multiple concentric blocks (for example, blocks of any shape sharing the same center) with the highest resolution block at the center of the user’s gaze point, thereby The resolution of adjacent blocks is lower than a preset value of full resolution, and the resolution of each block decreases in descending order from the user's gaze point. During the operation in which the peripheral detail mode is activated, the gaze point image frame includes a plurality of concentric blocks, and the block with the highest resolution is located on the periphery of the concentric blocks, so that the resolution of the second inner block is adjacent It is lower than a preset value of full resolution, and the resolution of each concentric inner block decreases in descending order. The header packet data may further include a transmission-mode toggle bit to enable raster-order mode and zone-order mode. During the operation in which the raster sequence mode is activated, the data transmission includes multiple line groups, which represent multiple rows of data from the concentric regions corresponding to the original image display sequence. During the operation in which the block sequential mode is activated, the data transmission includes the complete transmission of each of the multiple concentric blocks before the data transmission of adjacent blocks. The header packet data may further include a block quantity field defining the number of the concentric blocks; a block size field defining the horizontal and vertical dimensions of each of the concentric blocks; a block offset field defining the number The horizontal and vertical offsets associated with each of the concentric blocks; and multiple display parameters. The display parameters may include a character length segment, which defines a plurality of pixel bits transmitted in a timing cycle related to the drive controller circuit. Further, the display parameters may include x-offset size segments, which define multiple pixel bits in each Least Significant Bit (LSB) of each horizontal offset, and line group size segments, which must be defined simultaneously The maximum number of rows written. In addition, the display parameters may include a row time period, which defines multiple timing periods required for writing one row, and a dual-column drive mode indicator, which can simultaneously write two rows. In an embodiment of the invention, at least one of the blocks has a different center from one of the other blocks.

Whether defined in software or hardware, the gaze point display system 100 may further include gaze point display protocol logic to encode and decode gaze point image frames and gaze point modulation plane data. The gaze point display protocol logic Including: image agreement encoding 115, image agreement decoding 123, plane agreement encoding 125, and plane agreement decoding 133. In particular, the processor 110 may include an image agreement code 115. The drive controller 120 may include image protocol decoding 123 and plane protocol encoding 125. The one or more modulation devices 130 may include planar protocol decoding 133.

In operation, the processor 110 is coupled to receive image data about the image and the gaze area. In particular, the processor 110 may receive image input data from the one or more input data sources 105; and tracking data from the tracking logic 107 based on the user's retina and/or head position in real time. Optionally, the processor 110 may be coupled to receive gaze point data from one of the input data sources 105. In response, the processor 110 may generate a gaze point image frame based on the image data, where the gaze point image frame includes header packet data indicating two or more concentric blocks of different resolutions, so that each block is Compression is defined by multiple megapixels and the corresponding megapixel ratio. The processor 110 may transmit the gaze point image frame to one or more modulation devices 130 having a raster logic 150, wherein the raster logic 150 is coupled to the display circuit 140 including the pixel array 144. For example, the gaze point image frame with header packet data can be transmitted to the drive controller circuit 120, and the drive controller circuit 120 uses the conversion unit 124 to generate gaze point bit plane data; and based on the modulation format and the header The packet data uses the conversion unit 126 to convert the gaze point bit plane data into a modulation plane. The driving controller circuit 120 can transmit the modulation plane 127 to the one or more modulation devices 130. Then, the one or more modulation devices 130 may use raster logic 150 to decode and expand each modulation plane based on the header packet data and each corresponding giant pixel ratio to output the gaze point image to the display Or display pixel array. That is, for a gaze area with a reduced resolution, a single bit of the related plurality of macro pixels is copied to the line group of the display pixel array 144 based on the corresponding macro pixel ratio, wherein the corresponding macro pixel ratio is associated In each block.

塊可對應二次方之輸入像素在x與y上填補一顯示像素區2n x 2n （區塊Z0=1x1像素、區塊Z1=2x2像素、區塊Z2=4x4像素、區塊Z3=8x8像素）。 在一些實施例中，對於中心細部模式來說，區塊0在凝視中心係具有最高解析度，而其他同心區塊具有不同解析度。特別的是，區塊Z1圍繞區塊Z0且僅適用於區塊Z0之外的像素。區塊Z1的解析度低於區塊Z0的解析度。區塊Z2圍繞區塊Z1且具有的解析度低於區塊Z1的解析度。區塊Z3圍繞區塊Z2且具有的解析度低於區塊Z2的解析度。然而，對於周邊細部模式來說，區塊0仍係為最高解析度，但可以根據整體顯示器的長度與寬度而調整大小。區塊0當中的該些同心區塊具有的解析度低於區塊0的解析度。亦即，區塊Z1具有的解析度低於區塊Z0的解析度；區塊Z2具有的解析度低於區塊1的解析度；並且區塊Z3具有的解析度低於區塊Z2的解析度。注視點渲染模組112可根據其演算法判斷每個區塊的大小與位置，其中此演算法可包含總視場（total field-of-view）、光學系統失真（optical system distortion）、注視靈敏度（fovea acuity）、凝視追蹤器的公差/延遲（ tolerance/latency of gaze tracker）、運動速率（rate of motion）等因素。可選地，所述一或多個輸入數據源105或追蹤邏輯可向處理器110提供每個區塊的大小與位置。周邊細部模式可能更適用於使用干涉光操縱的顯示器；因此不依循著凝視追蹤。因此根據本發明的注視點協定、方法及系統減少了數據帶寬。在中心細部模式中，當凝視（gaze）朝向顯示器的邊緣移動時，該些區塊的邊緣可對齊使得區塊Z0與邊緣的偏移量（offset）為0。The system and method for gaze point display may include a center detail mode and a peripheral detail mode. For example, in one embodiment, there are four fixation zones (Z0~Z3), and each zone

Blocks can correspond to quadratic input pixels. Fill a display pixel area 2n x 2n on x and y (block Z0=1x1 pixel, block Z1=2x2 pixel, block Z2=4x4 pixel, block Z3=8x8 pixel ). In some embodiments, for the center detail mode, block 0 has the highest resolution in the gaze center, while other concentric blocks have different resolutions. In particular, the zone Z1 surrounds the zone Z0 and only applies to pixels outside the zone Z0. The resolution of the block Z1 is lower than the resolution of the block Z0. The block Z2 surrounds the block Z1 and has a resolution lower than that of the block Z1. The block Z3 surrounds the block Z2 and has a resolution lower than that of the block Z2. However, for the peripheral detail mode, block 0 is still the highest resolution, but the size can be adjusted according to the length and width of the overall display. The resolution of the concentric blocks in block 0 is lower than the resolution of block 0. That is, the resolution of block Z1 is lower than that of block Z0; the resolution of block Z2 is lower than that of block 1; and the resolution of block Z3 is lower than that of block Z2. degree. The fixation point rendering module 112 can determine the size and position of each block according to its algorithm, where the algorithm may include total field-of-view, optical system distortion, and gaze sensitivity (Fovea acuity), tolerance/latency of gaze tracker, rate of motion, etc. Optionally, the one or more input data sources 105 or tracking logic can provide the processor 110 with the size and location of each block. Peripheral detail mode may be more suitable for displays that use interference light manipulation; therefore, it does not follow gaze tracking. Therefore, the gaze point protocol, method and system according to the present invention reduce the data bandwidth. In the center detail mode, when the gaze moves toward the edge of the display, the edges of these blocks can be aligned so that the offset between the zone Z0 and the edge is 0.

There are several possible modes of operation: center detail pair (v.) peripheral detail; block sequence pair (v.) raster sequence. In at least one embodiment implementing the central detail mode, it is wrong to extend the block with a lower number outside the boundary of the block with a larger number. Therefore, if the block Z3 and the block Z2 have the same size and offset, the block Z3 will not be used. The largest block can be the same size as the display, so its offset is zero. However, if the display supports a partial display mode, the maximum block can be smaller than the display and a fill value will be used to hold the remaining blocks of the display.

In the peripheral detail mode, the size and position of these blocks can be constant, but it is not necessary. The fixation point protocol described here may allow dynamic or static size and offset. Applications that use the peripheral mode may not be able to use the large macropixels in block 3, but the concepts and methods described here are allowed. However, this capability depends on the capability of the display device to support it. If the size of the block Z2 or Z3 is 0x0, it is the same as the block unused. In at least one embodiment of implementing the peripheral detail mode, it is wrong to extend the block with a higher number to the outside of the block with a lower number.

In the block sequential mode, the data of each block will be sent completely before the data of the next block. If the block sequential mode is used, the data of block Z3 will be sent first; and the data of block Z0 will be sent last. The data of each block is sent according to the raster line order: horizontally from left to right, and then vertically from top to bottom. If the raster data is less than an integer multiple of words and row padding is enabled, padding will be performed to fill the last character; where each raster starts at a word boundary . For a raster that spans the blank area of another block, the data on each side of the block is packed together. Only add padding at the end of each raster.

In raster sequential mode, data is transmitted through line groups (refer to the detailed description below). Each line group starts with block 3 data, and then the first row of block 2 data. Therefore, the first row of each block (Z1, Z0) is sent. Depending on the format sequence in the line group, these lines are the second and third consecutive lines of each block. As each raster moves down in the display row, the data row is added, and the data of the highest numbered block is always the first one. Padding is added at the end of each block of data to become integer characters, so each new row and data block starts from a character boundary. Additional padding at the end of each line group may need to be added to meet the minimum line time requirements of the display.

Compared with the block sequential mode, the raster sequential mode can provide lower latency and minimal plane storage. Block order may be easier to implement and use less padding, but may result in more buffer space and delay. For example, if the block of block Z0 is located at the top of the display, the first row cannot be written until all blocks Z3, Z2, and Z1 are received. Even so, the implementation of the block sequence may not be able to keep up with the data rate of block Z0. The control of the block order can still use the size and row-times (see below) to calculate the total time to write to the display, but it will not be closer to the sending plane than this. If the display does not have double buffers for block data, it may be necessary to add extra time between the planes to prevent overlap.

The multiple parameters that control the data format to match the capabilities/limitations of the display may include: character size, x-offset size, line group size, line time, and dual column drive. The character size is the number of pixel bits transmitted with each timing of the interface (for example, a 64-signal DDR bus will send 128-bit characters). This may be the granularity on the horizontal block side. The X-offset size is the number of pixel bits of the least significant bit (LSB) of each horizontal offset, which is also called the step-size of the X offset. The offset size can be half of the character size to allow a block to be centered. The line group size represents the maximum number of lines that can be written at the same time.

The line time parameter represents the minimum number of characters in each line. That is, the number of timings required for the display to perform writing on one or more rows. The line group that only spans the highest block (with matching line group size) will only use one line time to complete this line group. Therefore, if X-resolution requires fewer characters, the data packet needs to be filled. If the line group spans three blocks, four line times will be required to execute the writing of this line group. Therefore, if the activated data has fewer characters, the data packet needs to be filled to four times the line time (number of characters). The row time of each block level can be unique. For high simultaneous line counts, this feature can be achieved by using interleaved line pulses. The row time of only data in block Z3 can be greater than the row time of data in block Z0.

Regarding dual-column driving, this system can allow two rows with unique data to be written simultaneously. This feature has two options: 1=half line group (the first half of the line group is driven by the first group of column drivers; this allows the problem of filling savings to be solved within a line group, but if the line group in the adjacent line group Each uses only one row time (1 row time), and adjacent line groups are not allowed to save time), 2=even/odd line groups (even line groups are driven by the first group of column drivers; when each line group writes a row Can save time, but need to calculate filling on two line groups).

The format of each modulation plane can contain a header, which includes: (a) the mode used, the number of blocks; (b) the size of each block in x and y; (c) in x The offset from each block on y; and (d) the display parameters/limits used. After the header packet data, modulation plane data may be included, and the data may be terminated by error detection protocols markers. For example, cyclic redundancy checking (CRC) may be added to the data sent to the one or more modulation devices 130.

More specifically, as shown in FIG. 1A, a display circuit (such as an LCoS circuit) 107 can generally include individually addressable (controllable) pixel elements (where each pixel is at least partially formed by electrodes formed on a semiconductor material). An array (XY) formed by liquid crystal materials or substances. The array size can generally be regarded as the upper limit of the resolution of the gaze point image 114, and the gaze point image will have two or more blocks, and at least one block has a resolution less than the maximum resolution of the displayed image. The array size can be, for example, 2048x2048, 4096x4096 or 6144x6144 pixel elements. The size of the gaze area may be, for example, 512x512 (1x1), 1024x1024 (2x2), 1536x1536 (3x3), 2048x2048 (4x4), 3072x3072 (6x6), and so on. For each of these examples of the size of the fixation area, the corresponding macro pixel array size is 512×512. In an embodiment of the present invention, the control of the pixel may include controlling the amplitude and/or delay (such as phase) of electromagnetic radiation (such as light) propagating through the pixel (such as transmission and/or reflection propagation); The nature of the gaze point image 146 displayed. For example, the modulation device circuit 130 may be configured to receive electromagnetic radiation (eg, light, such as laser light) and cause a phase shift of this electromagnetic radiation to produce a desired result.

In some embodiments, the system 100 may include a plurality of modulation devices 130, which may generate, for example, a color gaze point image 146 or an achromatic gaze point image 165 (which will be described in further detail with reference to FIG. 1B). In this embodiment, each modulation device 130 can be configured to control the color saturation of the projected image. For example, a system includes three modulation devices to control the red, green, and blue (RGB) of the projected image 146, respectively. ) Color saturation. In other embodiments, for example, a single modulation device may be used to generate a monochrome projection image or a color sequential image. The number of saturation levels for each pixel may be based on the number of levels defined by the gaze point image, and may be expressed in binary form, for example. For example, a 6-bit image data set may have a level of 26=64 per pixel per color.

Each resolution of each block of the fixation point image frame 103 can be defined by a unique (unique) giant pixel. A "macro pixel" as used herein generally refers to a group of two or more pixels/physical pixels, which are controlled by the modulation device circuit 130 to generate an image with a defined resolution. a part. For example, if the first block Z0 is defined as the block with the highest resolution of the modulation device circuit 130, the megapixels in the first block Z0 can be defined as 1x1 (both one to one on x and y) The corresponding relationship), which means that each pixel of the first block Z0 of the gaze point image frame 103 corresponds to a single physical pixel of the modulation device circuit 130. If the second zone Z1 is defined as having a resolution, the resolution is one-fourth of the resolution of the first zone Z0 (the correspondence between x and y is one to two), then The megapixels in the second zone Z1 can be defined as 2x2, which means that one megapixel in the second zone Z1 with reduced resolution corresponds to four physical pixels (2 physical pixels x 2 physical pixels) of the modulation device circuit 106 , The rest of the defined blocks can be deduced by analogy. Advantageously, since the coded gaze point image frame 103 includes a plurality of lower resolution blocks, the overall data size of the gaze point image frame 103 will be substantially smaller than the overall data size of the full-resolution image frame . Therefore, in addition to reducing the size of the memory/buffer for storing the gaze point image frame data, the bandwidth required for the communication interface between the processor circuit 110 and the drive controller circuit 130 is also reduced.

The drive controller circuit 130 is generally configured to receive the gaze point image frame data 103 from the processor circuit 110 and use the conversion unit 124 to generate gaze point bit plane data. The bit-plane data used here can include header information (similar to the header information in the gaze point image frame 103), an array of binary values of the macro pixels in each gaze point block (to control the corresponding display pixel The padding data between the giant pixel data) and/or the giant pixel data will be described in further detail below. In some embodiments, each frame system sequentially generates multiple bit planes. The binary values of the bit planes can be generated by, for example, pulse width modulation (PWM) and/or pulse frequency modulation techniques controlled by functions loaded by the drive controller 120. The series of binary values is based on the saturation value of each pixel of the gaze point image frame 103 and the selected modulation technique. The generated bit plane is an array of binary values, and each binary value corresponds to a pixel or a giant pixel. In an embodiment of the gaze point display system and method, each binary value corresponds to a physical pixel of the modulation device circuit 130. Optionally, each binary value corresponds to a macro pixel, which controls the collection of physical pixels in the modulation device circuit 130 defined by the size of the macro pixel. Advantageously, for the gaze point display system and method described herein, the bandwidth and data throughput requirements of the communication interface between the drive controller circuit 120 and the modulation device circuit 130 can be substantially reduced. In some embodiments, the drive controller circuit 120 may generate a header with one or more defined areas to define the size/position of the fixation point area, the size of the macro pixel, etc. This header may be formed as the first line or multiple lines of each frame/subframe set of multiple modulation planes or the first modulation plane of each modulation plane.

Generally speaking, this header packet data defines the number of multiple blocks and the size and location of each block. Regarding offsets, in some embodiments, the last block may have the same size as the display. In these cases, there is no offset between this block and its position (the offset relative to the origin, such as the upper-left (UL) will be (0, 0). In some embodiments, The largest block can be smaller than the entire display, so these offsets can be allowed to be non-zero. In such an embodiment, the display logic can fill the surrounding pixels with a predetermined value. The following shows an exemplary header, where the horizontal size (H -Sizes is a multiple of H-step-size (H-Size can also be a multiple of Word Size, where H-step-size is less than or equal to Word Size; please see below for details The horizontal offset (H-Offsets) is a multiple of H-step-size, and the vertical size (V-sizes) and offset are multiples of the line group size (usually four display lines). The shift is relative to the origin of the display (such as the upper left corner), and H-mpix and V-mpix are the size of the megapixels in the horizontal and vertical directions of the block (non-square megapixels can be used if necessary). All The fields are all in pixels of the display device:

In particular, the display driving controller 120 is used to transmit data to the display 146. Data is transmitted in units of characters, one of which is the block size of the data bus in a time sequence (for example, a 32-bit DDR bus can have a 64-bit block or character size (Wsize) =64)]. Each character is a horizontal line of bits from a block; in the same row, they all have the same resolution. For the content supported by the driver and the display device, the data can be transmitted in different sequences or formats. Therefore, in some embodiments, in order to define the format used to create a frame or plane, the header may also contain the following fields:

An exemplary configuration of the gaze point display system may include a gaze point image, the gaze point image has three blocks and each modulation plane is 640K bits, covering a four-megapixel display. The following table shows the relevant standards. Header packet data.

The following shows a line group passing through the middle of the image, which spans all three blocks. As shown below, there is a row of block Z2, which contains right segment data and left segment data (Z2A and Z2B), each segment has four characters (each character is 128 bits). In addition, there are two rows of block Z1, each of which contains two-character right and left segments (Z1Ar1, Z1Ar2, Z1Br1, Z1Br2). Further, there are four rows (Z0r1, Z0r2, Z0r3, and Z0r4) of the block Z0 data.

Therefore, the above line group can transmit a total of 22 characters in the following order:

In the second example, a line group is displayed, which represents data passing through blocks 2 and 3. As shown below, there is a row of block Z2, which contains the right and left data (Z2A and Z2B), and each segment has four characters (each character is 128 bits). In addition, there are two rows of the eight-character block Z1 (Z1r1, Z1r2).

Therefore, the above line group of this second example can transmit a total of 10 characters in the following order:

In the third example, a line group is displayed, which means that only data passing through block 3 is displayed. As shown below, there is a row of block Z2 (Z2r1), each row has sixteen characters (each character is 128 bits).

Therefore, the above line group of this second example can transmit a total of 5 characters in the following order:

Regarding the realization of the display, in order to meet the row timing of the low-resolution block 2, all four rows need to be written at the same time. These column drivers connect four consecutive rows and then simultaneously execute strobes for all four rows. Because these column drivers provide unique data for each row, the strobe pulses of these four rows are implemented separately when the block 0 data is written.

In some displays, when only data is written to block 2, LSS=4, Vmpix=4, and Hmpix=4, and the huge pixels reduce the data bandwidth so that the same amount of padding can be added to meet the minimum line timing. These displays can solve this problem by using a multi-drive architecture with MD=2ps; data is buffered for eight rows; there are 2x high-resolution (Hres) column drivers. The first 4 rows are connected to the same column driver; the 4 rows of the second group are connected to different column drivers; the rows of the third group are connected to the same column drivers as the first, and so on. This allows the eight rows to be written together and provides 2x the data time applied to the writing time of one row. In the above example 1, if LSS=4, MD=2ps and n=8, no padding characters are required under any circumstances.

It should be understood that the components of the exemplary operating environment 100 are exemplary, and there may be more or fewer components in various configurations. It should be understood that the operating environment may be a part of a distributed computing environment, a cloud computing environment, a client server environment, and so on.

Please refer to FIG. 1B, which illustrates a system schematic diagram of a fixation point electromagnetic radiation modulation with a grayscale device circuit according to some embodiments. Similar to the gaze point electromagnetic radiation modulation system 100 of FIG. 1A, the gaze point display system 160 of FIG. 1B may include a processor 110, a grayscale device circuit 162 and a display 165. The processor circuit 110 has a block definition module 111, a gaze point rendering module 112, a gaze point image memory 114, and an image protocol encoding module 115. The processor circuit 110 is generally used to receive one or more input data The input of the source 105 is used to generate a gaze point image frame 103. The gaze point image frame 103 has a header packet data defining two or more concentric blocks of different resolutions, where each block is compressed to be multiple giants. Definition of pixel and corresponding megapixel ratio. In some embodiments, the input from one or more input data sources 105 includes one or more images and associated gaze block data. In some embodiments, the processor 110 may be included in a host computer (not shown in the figure), so that the host computer transmits the gaze point image frame 103 to the grayscale device circuit 162 to be associated with the display 165. The grayscale device circuit 162 may include image protocol decoding logic 163, display circuit 170, raster logic 180, and memory 164. The display circuit 170 may include a control unit 172 and a pixel array 174; and the raster logic 180 may include a line group collection logic 182, a direct write logic 184, a row buffer 186, and a row-to-column 188. Of course, it should be understood that the grayscale device circuit 162 and/or the processor circuit 110 may also include other conventional and/or dedicated circuits and/or logical structures, such as frame buffer memory/cache, sequential circuits, vertical /Horizontal scan line circuit, processor circuit, etc. The gaze point image (such as the gaze point image or the direct gaze point image) stored in the memory 114 is output to the display 165 or rendered on the display 165. The gaze point image may include multiple resolution blocks Z0, Z1 , Z2, Z3,… ZN, where each block has a different resolution. For example, as shown, these blocks can be generated using the center detail mode, where the block Z0 with the highest resolution tracks the user's gaze direction, and the other blocks (Z1, Z2, Z3,...ZN) The resolution of is lower than that of block Z0, so that the resolution of each block decreases in descending order from the user’s gaze point.

The method of FIGS. 3A and 3B will detail the line group collection circuit 182 as a part of the memory read from the block buffered data or the received raster sequence data in real time. In some embodiments, the display device may provide direct write logic 184 to be able to write a part of a row corresponding to a block without affecting other parts of the row. This feature allows to write each block directly in block order without having to collect giant pixel data from different blocks in the line group. However, using the multi-line writing time to perform writing on all parts of the row may be the limiting factor of this embodiment.

FIG. 2A is a method flowchart of a method 200 for gaze point display according to some embodiments. In some embodiments, the method and protocol for displaying gaze points includes receiving in step 210 eye tracking data associated with generating gaze point block definitions. For example, the processor may receive image input data and tracking data based on real-time user retina and/or head position. In response, the method may further include generating a rendered gaze point image based on the image data and parameters, wherein the parameter defines a gaze point block (in step 215) according to the gaze point rendering technology, wherein the rendered The fixation point image includes macro-pixel image data, which corresponds to two or more concentric blocks with different resolutions and corresponding macro-pixel ratios. In response, the method may further include generating a gaze point image frame (in step 220) based on the rendered gaze point image and the agreement selected parameters, wherein the gaze point image frame includes header packet data , Which indicates two or more concentric blocks with different resolutions, so that each block is defined by the ratio of multiple macro pixels to the corresponding macro pixels.

FIG. 2B is a flowchart of a method 230 for generating a gaze point modulation plane by the drive controller circuit 120 of FIG. 1A according to some embodiments. In step 233, according to the block or line group selected by the header parameters, the gaze point image frame is decoded into different blocks, rows and columns of headers and macro pixel data (such as effective macro pixel data). Multiple control parameters. In response, the method may further include processing the image pixel data in step 235 to convert the pixel data into a format compatible or more compatible with the display, which may involve dithering, scaling/attenuation, and/or segmentation. For multiple bit planes, and selectively store the results in memory. In response, in step 240, the system can read data from the memory according to a modulation format to generate a modulation plane. In response, in step 244, the modulation plane data can be encoded into a modulation plane format/protocol with header data according to a selected plane protocol, and the result can be transmitted to one or more modulation planes. Device. In response, as a loop operation 248, this method determines whether the modulation format is complete. This method will repeat steps 240 and 244 until the fixation point image frame is completed, and then it will wait for the start of the next image frame to start again. Start execution from step 233.

2C is a schematic flow diagram 250 of a method or procedure for writing gaze point modulation plane data into a display pixel array in the gaze point modulation display device 130 of FIG. 1A according to some embodiments. In step 252, the method may include parsing the header packet data from the modulation plane and identifying the gaze area information. In response, according to step 254, the method can write the data into the memory 258 or directly process the data to write it into the pixel array. According to step 260, the method may further include an address management function, using the fixation area parameter to determine the size and sequence of the corresponding line group macro pixel data, which may include padding. In response, step 264 can collect macro pixel data by reading the memory 262 or directly input the data, according to which the macro pixel data is expanded and combined into line buffered display data according to the corresponding macro pixel ratio and block offset ( As shown in Figure 4), and transferred to a row queue for writing to the pixel array, which can simultaneously write the same data into one or more rows according to the corresponding macro pixel ratio. Further, in step 268, the method may include advancing control counters and indexes for the next line group. As a loop operation, this method may include repeating steps 260, 264, and 268 until each line group of the modulation plane is written into the pixel array in step 269.

FIG. 2D is a schematic flow diagram 270 of a method or procedure for writing gaze point image frame data into the display pixel array of the gaze point grayscale display device 162 of FIG. 1B according to some embodiments. In step 274, the method may include parsing the header packet data from the gaze point image frame, and identifying the gaze area information and valid macro pixel data. In response, according to step 276, the method can write data to the memory 278 or directly process the data to write the pixel array. This method may further include an address management function according to step 280, which uses the gaze area parameter to determine the size and sequence of the corresponding line group macro pixel data, which may include padding. In response, step 284 can collect macro pixel data by reading the memory 282 or directly input the data to expand and combine the macro pixel data into a row of buffered display pixel data according to the corresponding macro pixel ratio and block offset. Figure 4), and transferred to row to column for writing to the pixel array, the same data can be written into one or more rows at the same time according to the corresponding macro pixel ratio. Further in step 290, the method may include advancing the control count and indexing the next line group or the next subframe. As a loop operation, the method may include repeating steps 280, 284, and 290 until each line group of the image frame has been written into the pixel array in step 295.

3A is a schematic flowchart of a method or process of writing a modulation plane of gaze point data into the pixel array of the gaze point modulated display device 130 of FIG. 1A in the center detail mode operation according to some embodiments, where there is Four blocks. Generally speaking, the operation 300 includes a process of performing decoding on the display device and operating the incoming modulation plane with the center detail (Z0 in the center). In particular, the flowchart 300 illustrates the operation of the modulation plane state diagram with the center detail of the four-block fixation point image. For example, each of the four blocks uses a square megapixel whose size is equal to the power of two of the block (for example, the block 3 mpix size is 2 ³ =8 or 8x8).

In some embodiments, the method 300 for writing the modulation plane of the gaze point data into the pixel array of the gaze point display device may include parsing the header packet data in step 301 to detect the number of blocks, the size of the horizontal block, Line group size, line time, character size and x offset size. For example, the one or more modulation devices 130 may wait for the modulation plane to be sent to parse the header packet data. You can use strobe or data signature to indicate the start of a new header packet data/modulation plane.

In an embodiment of the present invention, the plane decoding logic 133 can retrieve all header fields to control multiple modes and parameters of the current modulation plane. These can be used in the following decision-making steps.

The method of writing the modulation plane of the gaze point data into the pixel array of the gaze point display device may also include detecting whether the transmission mode trigger bit is set to enable a raster sequential mode in the decision step 302. If there is a raster sequence mode modulation plane, start from the first line by initializing the counter/index, and process the incoming data as a line group in a predetermined data sequence (at step 310). In response to not detecting that there is a raster sequential mode, the method can continue to branch to the block sequential mode in step 304. At this stage, two separate processes begin to handle writing (380) and reading (306) zone buffers. It should be noted that the read side will be delayed until the end of the write sequence or after the end of the write sequence, so that when a read is required, the data of block Z0 (with the last block 0) is ready Complete (described with reference to the timing diagram 500 of FIG. 5). The buffer read time can usually be longer than the buffer write time because any required padding time is added to the large group of giant pixel lines used for array writing. In the block sequence mode, this method can include writing the corresponding block data of the incoming data into the corresponding block buffer {Z(n-1),..., Z3, Z2, Z1, Z0} (in step 380~388), and wait for a predetermined time of reading delay in step 306.

At the end of the read delay in step 306, the method may include switching a read pointer corresponding to each respective block buffer, and starting a read data stream to match the order required by the raster order ( Write the line group to this array). Since the internal buffer read path is not limited by the IO bandwidth, some devices can optimize the timing in the block sequence mode, and can read block Z0 data through a wider and/or faster bus to match the smallest row write Input timing to compensate for the filling timing added to the large giant pixel line group. Therefore, compared with the raster sequence, the sequence of the array write line group in the block sequence is different. Since the need for filling data has been eliminated, this step makes the writing speed of the block sequential modulation plane faster than the raster sequence.

At the end of the read delay and in response to detecting the raster sequential mode, the method may include identifying the first row of the data line group in step 310. The line and line group count will continuously track the corresponding position on the display related to the size and position of each block, so as to know whether the current line group crosses or intersects with each block. Some displays can provide a function. This function defines the size of the modulation plane to be smaller than the size of the overall display, and then places the active data of the modulation plane in the display area with an offset. This initialization step 310 will consider such an offset.

The method may further include detecting whether a high-resolution block (Z0) of a plurality of concentric blocks exists in the identified data line group in a decision step 320. In particular, the decoding logic will detect whether this line group contains a part of the block Z0. In other words, whether the line selected from the line group crosses or intersects the block Z0. Specifically, the current position of the indicator is compared with the size and offset of the block Z0 and the global offset. If the answer is yes, this line group will also intersect all the blocks above, and block Z0 is in the center. That is, by confirming that block Z0 is detected, there is no need to detect whether other blocks exist in this row. The method 300 can continue to process the line group data in steps 322 and 324.

In response to not detecting the existence of a high-resolution block, this method may include detecting the next consecutive block (Z1, Z2, Z3, ... Z(n-1)) in decision steps 330, 340, and 350 Whether it exists in the identified data line group until a block is detected. In response to detecting a block, this method can include expanding the identified data line group based on the number of blocks, horizontal block size, line group size, x offset size, line time, and character size; In steps 322, 332, 342, and 352, the expanded data is stored in a row buffer. Further, this method can include when individual blocks {Z0, Z1, Z2, Z3... Z(n-1)} are detected, the row buffer is transferred to row to row, where k rows are written Enter 2 (r-1) times, r = {n, n-1, n-2, n-3, …1}, k = {1, 2, 4, 8, …2(n-1)}, n is the number of blocks (in steps 324, 334, 344, and 354).

Regarding the expansion of this line group, since the header packet data contains the details of the horizontal size of each block, the decoding logic can identify how many characters will be received for each block. Taking the particular example shown in FIG. 3A for example, assuming the JIT sequence, where the block Z0 is the last of the row and the data of the block Z3 is the first of the row. In step 322, for the block Z0 data, each character is horizontally expanded eight times, and the final result of combining/merging multiple characters is stored in the row buffer with the correct offset. After this same example, the next one will be block Z2 data, each character will be expanded horizontally four times and stored in the line buffer with the correct offset. Then, the block Z1 data is horizontally expanded twice and stored in the row buffer with the correct offset. Furthermore, the data of block Z0 will be the last one, and will be directly stored in the row buffer with the correct offset. In step 324, a row is written eight times. In addition, the method 300 may include transferring the row buffer to the row to column. It is possible to start row writing to the array before all eight rows are in the queue. Each row can be written separately, requiring a total of eight write cycles. The write cycles are spaced to match the minimum line time for one line at a time, (x1 mode), so queues are needed.

After the example of block Z0 data detection, then the data during the detection period of other respective blocks Z1~Z3 (in steps 330, 340, and 350), in steps 322, 332, 342, and 352, In a similar way, the block data of each individual block is expanded and merged in the row buffer. That is, each character of the block Z0 data is expanded eight times horizontally, and the final result of combining/merging multiple characters is stored in the row buffer with the correct offset. After this same example, the next one will be block Z2 data, each character will be expanded horizontally four times and stored in the line buffer with the correct offset. Then, the block Z1 data is horizontally expanded twice and stored in the row buffer with the correct offset. Furthermore, the data of block Z0 will be the last one, and will be directly stored in the row buffer with the correct offset. During the detection of block Z1, the two rows are written four times in step 334. During the detection of block Z2, in step 344, four rows are written twice. During the detection of block Z3, in step 354, eight rows are written once.

In addition, the method may include retrieving the next data line group in step 360, and repeating detection, expansion, storage, transmission, and retrieval until the retrieval of each data line group is completed (steps 362 and 320-360). That is to say, for example, this process continues, for the remaining seven rows of this line group (where the line group is eight rows), receive data from the blocks Z0~Z2, and transfer these to the line queue. Therefore, in the decision step 362, the method can detect whether the current row is the last row. If not, the process will proceed to step 320 and process the next line group. If the current line is the last line, the method continues to execute decision step 364 to detect whether the raster mode is enabled. When the raster sequence mode is enabled, the method step will branch back to check the next modulation plane (in step 301). If not, there is a block sequential mode, and the process ends in this thread. In some embodiments, another thread writes buffered data and returns to detect the next header packet data. Optionally, this other thread may currently be writing buffer data of the next modulation plane. In some embodiments, the A/B index switch on the buffer reading side may be performed when the reading sequence ends instead of starting.

In steps 380, 382, 384, 386 and 388, the data from the respective blocks Z3, Z2, Z1 and Z0 are written to the corresponding block buffers (block 3 buffer, block 2 buffer, block 1 Buffer and block 0 buffer). Some macro pixel rows will be the full block width; other macro pixel rows that intersect another adjacent block may be shorter. The padding mode can be enabled to adjust data control when matching buffer addressing is required. The expected number of characters in each block data is calculated by controlling the size and mode. When writing each block of data, the method can include steps 382, 384, and 386 to move to the next block of data. In step 388, the method ends the block sequence processing by switching the buffer A/B write side indicator and returning to wait for more modulation planes with header packet data.

3B is a method or process of writing the gaze point data modulation plane in the peripheral detail mode operation in the pixel array of the gaze point modulation display device 130 of FIG. 1A according to some embodiments, in which there are four regions Piece. Similar to the example in FIG. 3A, the flowchart 400 shows the operation of the modulation plane state diagram. The difference is that the operation mode is that the display device can decode and process the incoming modulation plane according to the surrounding details, where the block Z0 is located The periphery of this display. For example, four block gaze point images will be displayed, where each of the four blocks uses a rectangular giant pixel with a size equal to the power of two of the block (for example, the block 3 mpix size is 23 =8 or 8x8).

In some embodiments, the method 400 for writing the modulation plane of the gaze point data into the pixels of the display array may include in step 401, parsing the header packet data to detect the number of blocks, the size of the horizontal block, the line Group size, line time, character size and x offset size. For example, the one or more modulation devices may wait for the modulation plane to be transmitted to parse the header packet data. The strobe or data signature can be used to indicate the start of a new header packet data/modulation plane. Effectively, this plane decoding logic can retrieve all header fields to control the mode and parameters of the current modulation plane. These can be used in the following decision-making steps.

In the peripheral detail mode, the method of writing the modulation plane of the gaze point data into the pixel array of the gaze point display device may also be included in the decision step 402, detecting whether the transmission mode trigger bit is set to enable the raster sequence mode . If there is a raster sequence mode modulation plane, start from the first line by initializing the counter/index, and process the incoming data into line groups in a predetermined data sequence (in step 410). In response to the raster sequential mode not being detected, the method may branch in block sequential mode in step 404. At this stage, two separate processes will begin to handle write (480) and read (406) block buffers. It should be noted that the read side will delay until the end of the write sequence or after the end of the write sequence, so that when it needs to be read, the data of block Z0 (with the last block 0) is ready to be completed (refer to The timing diagram 500 of FIG. 5 is described). The buffer read time can usually be longer than the buffer write time because any required padding time is added to the large group of giant pixel lines used for array writing. In the block sequence mode, this method can include writing the corresponding block data of the incoming data into the corresponding block buffer {Z(n-1),..., Z3, Z2, Z1, Z0} (in step 480~486), and wait for a predetermined time of reading delay in step 406.

At the end of the read delay in step 406, the method may include switching the read index corresponding to each individual block buffer, and starting to read the data stream to match the order required by the raster order (write line group Array). Since the internal buffer read path is not limited by the IO bandwidth, some devices can optimize the timing in the block sequence mode, and can read block Z0 data on a wider and/or faster bus to match the smallest row write Input timing to compensate for the filling time added to the large giant pixel line group. Therefore, compared with the raster sequence, the array write line group timing of the block sequence is different. Since the time required to fill the data has been removed, this step can make the writing speed of the modulation plane of the block sequence faster than that of the raster sequence.

At the end of the read delay and in response to detecting the raster sequential mode, the method may include identifying the first row of the data line group in step 410. The line and line group count will continuously track the corresponding position on the display related to the size and position of each block, so as to know whether the current line group crosses or intersects with each block. Some displays can provide a function. This function defines the size of the modulation plane to be smaller than the size of the overall display, and then places the active data of the modulation plane in the display area with an offset. This initialization step 410 will consider such offset.

The method may further include in the decision step 420, detecting whether the lowest resolution block (Z3) in the center of the concentric blocks in the peripheral detail mode exists in the identified data line group. In particular, the decoding logic will detect whether the line group includes a part of the block Z3. In other words, whether the line selected from the line group intersects the block Z3. Specifically, the current position of the indicator is compared with the size and offset of block Z3 and all global offsets. If the answer is yes, then this line group will also intersect all the upper blocks, and block Z3 is located in the center. In other words, by confirming that block Z3 is detected, there is no need to detect whether other blocks exist in the row. The method 400 can continue to process the line group data in steps 422 and 424.

In response to not detecting the existence of the lowest resolution block, the method may include detecting whether the next continuous block (Z2, Z1, Z0) exists in the identified data line group in steps 430, 440, and 450 Until a block is detected. In response to detecting a block, the method may include in steps 422, 432, and 442, determining the number of blocks, horizontal block size, line group size, x offset size, line time, and character size in steps 422, 432, and 442. The data line group is expanded; based on this x offset size, the expanded data is stored in a row of buffers. Further, the method may include in steps 424, 434, 444, and 454, buffering the row to row-to-column, where each row is written 2 (n-1) times (or, for example, 23 times = 8 times) , Where n is the number of blocks (here, for example, n-1=3).

Regarding the expansion of the line group, since the header packet data contains the details of the horizontal size of each block, the decoding logic will be able to identify how many characters are received for each block. For the particular example shown in FIG. 3B, assume the JIT sequence, where the block Z3 is the last in the row, and the data in the block Z0 is the first in the row. In step 422, for the block Z3 data, each character is horizontally expanded eight times, and the final result of the combined/combined character is stored in the row buffer with the correct offset. After the same example, the block Z2 data will be the next one, and each character will be expanded horizontally four times and stored in the row buffer with the correct offset. Then, the block Z1 data is horizontally expanded twice and stored in the row buffer with the correct offset. Further, the data of block Z0 will be the last one and will be directly stored in the row buffer with the correct offset. In step 424, a row is written eight times. In addition, the method 400 may include transferring the row buffer to the row to column. Row writes to the array can begin before all eight rows are in the queue. Each row is written individually, for a total of eight write cycles. The write cycles are spaced to match the minimum line time for one line at a time, (x1 mode), so queues are needed.

After the example of block Z3 data detection, the data during the detection period of other respective blocks Z2~Z1 (in steps 430, 440, and 450) are followed in steps 422, 432, and 442, similarly In this way, the block data of each individual block is expanded and merged in the line buffer. That is, each character of the block Z2 data is horizontally expanded four times, and the final result of combining/merging multiple characters is stored in the row buffer with the correct offset. After this same example, the next one will be block Z2 data, each character will be horizontally expanded twice and stored in the line buffer with the correct offset. Then, the block Z1 data is expanded once horizontally and stored in the row buffer with the correct offset. Further, the data of the block Z0 will be the last one, and will be directly stored in the row buffer with the correct offset (as shown in step 452 through data (pass through data)). During the detection of blocks Z2, Z1, and Z0, a row is written eight times in steps 424, 434, 444, and 454, respectively.

In addition, the method may include capturing the next data line group in step 460, and repeating the steps of detecting, expanding, storing, transmitting, and capturing until each data line group is captured (steps 462 and 420~ 460). That is, for example, when data from the blocks Z3 to Z0 are received for the remaining seven rows of the line group (where the line group=8 rows) and these are transmitted to the row to column, the process continues. Therefore, in decision step 462, the method can detect whether the current row is the last row. If not, the process will execute step 420 and process the next line group. If the current line is the last line, the method executes decision step 464 to detect whether the raster mode is enabled. When the raster sequence mode is enabled, the method step will branch and return to confirm the next modulation plane (at step 401). If not, there is a block sequential mode and the process ends in this thread. In some embodiments, another thread writes buffered data and returns to detect the next header packet data. Optionally, this other thread may currently be writing buffer data of the next modulation plane. Some embodiments may perform the switching of the A/B index on the buffer reading side at the end of the reading sequence and not at the beginning.

In steps 480, 482 and 484, the data from the individual blocks Z3, Z2, Z1, and Z0 are written into the corresponding block buffers (block 3 buffer, block 2 buffer, block 1 buffer, and block 0 buffer). Part of the megapixel rows will be full block width; other megapixel rows that intersect another block may be shorter. The padding mode can be used to adjust the data control as needed to match buffer addressing. Calculate the expected number of characters in each block of data according to the size and mode control. When writing each block of data, the method may include moving to the next block of data in steps 482, 484, and 486. In step 488, the method ends the block sequence processing by switching the indicator on the write side of the buffer A/B, and waits for more modulation planes with header packet data.

Please refer to FIG. 4A, which is a schematic diagram 490 of a display with four blocks in the center detail mode, indicating that the line group segment 494 of eight rows crosses the display center so that the line group crosses all four blocks. This figure is applied to the Just-In-Time (JIT) data sequence or line group collection function of the raster sequence protocol. The steps associated with expansion and combination correspond to the step 322 outlined in FIGS. 4B and 4C.

Please refer to FIG. 4B, which illustrates a schematic diagram of the multiple steps of the macro-pixel data expansion in step 322 in the method of FIG. 3A according to some embodiments, which shows the content of the macro-pixel data block and the content of the line buffer. In raster sequential mode, the method includes matching using JIT sequence to write immediately. In the block sequential mode, the method may include internal buffer reads to collect data for line group writes. For the given example, there are one row of block 3 megapixels, two rows of block 2 megapixels, four rows of block 1 megapixels, and eight rows of block 0 megapixels. As shown in step 1, the method may include horizontally expanding the megapixels of block 3 eight times. The separate A and B areas represent the left and right sides of the block in the row of the current line group. This data is written into one line of buffer with block 3 offset (z3HoffsetA&B). As shown in step 2, the method may include expanding block 2, in which row 1 megapixels are copied horizontally with block 2 offset (z2HoffsetA&B) four times to the right and left segments of the row buffer. As shown in step 3, the method may include expanding block 1, in which row 1 megapixels are copied horizontally with block 1 offset (z1HoffsetA&B) twice to the right and left segments of the row buffer. As shown in step 4, the method may include writing to block 0, in which row 1 pixels are copied to the row buffer with a block 0 offset (z0Hoffset). The method may include using a steering horizontal offset (Hoffset) to copy the row buffer to row to column, where the outer pixels are filled with a predetermined fill value. As shown in step 5, the method may include writing the second row (row 2) of block 0 into the row buffer with a block 0 offset (z0Hoffet). Then, use the steering horizontal offset (Hoffset) to copy the row buffer to the row-to-column, and fill the outer pixels with the same predetermined filling value.

Referring now to FIG. 4C, it is a schematic diagram of the multi-step continuation of the expansion of the giant pixel data of FIG. 4A according to some embodiments. As shown in step 6, the method may include horizontally extending the second line of the secondary block 1 to the right and left segments of the line buffer by the block 1 offset (z1Hoffset). As shown in step 7, the method may include writing the third row of block 0 into the row buffer with a block 0 offset (z0Hoffet). As in step 8, the method may include writing the fourth row of block 0 into the row buffer with a block 0 offset (z0Hoffet). As in step 9, the method may include expanding the second row of block 2, where the macro pixels of row 2 are copied horizontally four times to the right and left segments of the row buffer with a block 2 offset (z2Hoffset). As shown in step 10, the method may include expanding the third row of block 1, where the macro pixels of row 3 are copied horizontally to the right and left segments of the line buffer with a block 1 offset (z1Hoffset). As shown in step 11, the method may include writing the fifth row of block 0 into the row buffer with the offset of block 0 (z0Hoffet). As shown in step 12, the method may include writing the sixth row of block 0 into the row buffer with the offset of block 0 (z0Hoffet). As shown in step 13, the method includes horizontally extending the fourth line of block 1 to the right and left segments of the line buffer by the block 1 offset (z1Hoffset). As shown in step 14, the method may include writing the seventh row of block 0 into the row buffer with the offset of block 0 (z0Hoffet). As shown in step 15, the method may include writing the eighth row of block 0 into the row buffer with the offset of block 0 (z0Hoffet).

Please refer now to FIG. 5, which is a schematic diagram 500 of a block sequential image frame or a timing diagram of a modulation plane according to an embodiment of the present invention, which shows the block buffer writing and reading sequence. The following is based on the embodiment with the modulation plane of the modulation display device, and can be similarly applied to the embodiment of the image frame of the grayscale display device. The buffer of each block is divided into A and B halves/sides representing two consecutive planes (or frames); the data of one plane should be located on one half/side. Assuming that the writing index selects the A side, the data received from the first plane is written to the A side of the buffer block by block. This header indicates the delay time starting from the A side of the first plane, from the header to the start of buffer reading. In the reading process, the data of each line group will be sequentially collected from all blocks according to requirements (line group 0 starts first, then line group 1, and so on), and then they are written into the array. As long as the write of the block 0 line group is maintained before the read, the read can overlap with the end of the write sequence. The next plane written to the B side can start before the A side read is completed. At the end of array writing, some displays require some manipulation to apply data to active pixel elements.

As used herein, the term module may describe a given functional unit that can be executed according to one or more embodiments of the invention. As used herein, the module can be implemented in the form of hardware, software or a combination thereof. For example, one or more processors, controllers, application-specific integrated circuits (ASICs), programmable logic arrays (PLAs), programmable array logic (PALs), complex programmable logic devices (CPLDs), Field programmable logic arrays (FPGAs), logic components, software routines or other mechanisms are used to form modules. In the implementation, the various modules described herein may be implemented as discrete modules, or the functions and features described herein may be partially or fully shared among one or more modules. In other words, it will be obvious to those of ordinary knowledge in the field that after reading this article, the various features and functions described herein can be implemented in any given application, and can be combined and replaced with one or more independent Implemented in the module. Even if various features or functions can be separately described or claimed as separate modules, those skilled in the art will understand that these features and functions can be shared in one or more common software and hardware components, and such descriptions are not Require or imply the use of separate hardware or software components to achieve features and functions.

It should be understood that the method described herein can be executed by a digital processing system, such as an existing general-purpose computer system. Alternatively, a computer designed or programmed to perform only a single function for a specific purpose can also be used. FIG. 6 shows an example of a computing device 600 that can implement the embodiments described herein. According to some embodiments, the computing device of FIG. 6 may be used to perform an embodiment of the gaze point image display function. The computing device 600 includes a central processing unit (CPU) 602, which is coupled to a memory 604, a video driver 607, and a mass storage device 608 through a bus 606. The mass storage device 608 represents a permanent data storage device, and in some embodiments may be, for example, a local or remote floppy disc drive (floppy disc drive) or a fixed disc drive (fixed disc drive). In some embodiments, the mass storage device 608 can implement backup storage. The memory 604 may include read-only memory, random access memory, and so on. In some embodiments, applications residing on the computing device can be stored in a computer readable medium, or accessed through a computer readable medium. The computer readable medium is, for example, memory 604 or large capacity Storage device 608. Applications (Applications) can also be in the form of modulated electrical signals, accessed through other network interfaces of network modems or computing devices. It should be understood that, in some embodiments, the central processing unit 602 may be implemented by a general-purpose processor, a special purpose processor, or a specially programmed logic device.

The display 612 communicates with the central processing unit 602, the memory 604 and the mass storage device 608 through the video driver 607 and the bus 606. The display 612 is used to display any visual tools or reports related to the system described herein. The input/output device 610 is coupled to the bus 606 for transmitting the information in the command selection to the central processing unit 602. It should be understood that the input/output device 610 can be used to transfer data to and from external devices. The central processing unit 602 can be defined to perform the functions described herein to enable the functions of FIGS. 1A to 4B. In some embodiments, the code that realizes this function can be stored in the memory 604 or the mass storage device 608 for execution by a processor such as the central processing unit 602. The operating system on the computing device may be iOSTM, MS-WINDOWSTM, OS/2TM, UNIXTM, LINUXTM, VXWORKSTM or other well-known operating systems. It should be understood that the embodiments described herein may also be integrated with a virtual computing system.

The following images, image-related and/or other data characteristics represent data, protocol parameters, components, and/or header components used to transmit or send images, image-related and/or other data to the display.

Horizontal step size (H-step-size) (horizontal size/offset step size)

H-step-size represents the horizontal offset or the increment of the horizontal size. When defining the size or offset, the block is defined by a multiple of H-step-size. If the value of H-size or H-offset is not a multiple of H-step-size, it will usually cause an error because it indicates that there is a conflict between the format of the transmitted data and the data decoding expected at the receiver. In some embodiments, two different H-step-size parameters may be used (for example, H-size has a more precise step size than H-offset). For example, this value can be selected based on the decoding and merging capabilities of the display device for different blocks of giant pixel data and minimizing the display multiplexing logic. The system control algorithm applied by the processor 110 can adjust/move the size/offset of each block to match these step increments. In some implementations, the size of the block may need to be one order larger than the area of interest to ensure that the area of interest is included in the block to achieve a small gaze point shift. For example, given H-step-size=32 and Z0Hsize=512, then Z0Hoffset can be {0, 32, 64, 96,... 32X (multiple of 32)}. The area of interest with eye tracking tolerance used to calculate Z0Hsize can be greater than or equal to 480 (480 + 32 = 512 is a multiple of 32). A smaller step size reduces the amount of additional high-resolution bandwidth. In some embodiments, H-step-size may be a multiple of the horizontal size of the largest megapixel to prevent H-size or H-offset from being defined as a fraction of megapixels.

Data Order

According to the present invention, the gaze point display system with gaze point protocol or method includes two data order options: block order or raster order. In the block sequential mode operation, all data of a block is first sent, and any cutout pixels in the block are removed. If multiple blocks are used, then all the data of the second block will be sent, and so on until the last block. Further, in the operation of the block order mode, the order of the blocks may be different. In other words, the block data with the highest resolution Z0 can be the first or the last. Compared to the raster sequence mode, the block sequence can have the least amount of padding because it can avoid the timing associated with the display device. In the block sequential mode, one or more modulation devices may be required to buffer data, and then appropriately collect data from a row (or group of rows) in each block before sending or writing.

When using raster sequential mode operation, multiple characters are sent in the line data group corresponding to the line group size (LSS) output line. For example, LSS can be four (LSS=4). However, LSS is also configurable (described in further detail below). In the first-line group, there are at least two options available: block group order (with the last block 0) or just-in-time order (JIT). JIT first collects all the data required for the first row (with the last block 0 data), then collects the new data for row 2, and then collects the new data for row 3, and so on, making buffering the simplest.

In raster order and block order mode operation, data can be moved line by line from the top/start to the bottom/end of the image in a raster-like manner, which may be related in a top-down order or a bottom-up order于Display device. In raster sequence mode, the display device keeps track of the number of lines and line groups; then compares the current line/line group with the header information of each block to determine whether the current line/line group intersects one or more blocks . The raster logic 150 checks the number of corresponding characters and the format of the line group according to the header parameters (see the details of FIG. 3A and FIG. 3B of the present invention for details).

The one or more modulation devices 130 can flip or reverse data processing in a horizontal or vertical direction. In addition, the modulation device 130 can manipulate the entire image or at least a part of the image through a certain number of pixels to match system requirements. These functions may be orthogonal to the fixation technology and work together without interference.

The data sequence of the image frame format may be different from the data sequence of the modulation plane format. In some embodiments, the host processor may prefer the block sequential format, and the display device may prefer the raster sequential format. Regarding these two interfaces as a pair, if the display device accepts the block order, it can be a block-to-block. If the drive can be converted from one to another, it can be block to raster. If the host processor can output the raster sequence, it can be raster to raster.

Padding

The gaze point display system and method disclosed herein may include four different types of padding: padding per block, padding per row, padding per group, and minimum row time padding. Different embodiments can implement different forms of padding. In at least one embodiment, the raster sequential format can enable all four padding types. In another embodiment, the block sequential format may only enable padding per block. Each type can be activated/required separately, so one or all types of padding can be used in the same frame or plane.

Per-zone padding

The conversion between data of different resolutions from different blocks may require different controls to expand the data and/or change the target. Therefore, blocks of different resolutions should not be mixed in the same transmission character. As a result, when the size of the data block is not a multiple of the character size (Wsize), the gaze point display system and method disclosed herein realize the filling of the last character of a block. Depending on other configuration options (such as character size, block size, and bit depth), other padding may not be needed to meet this limitation. When this padding is enabled in flat format, dummy padding pixels will be added instead of pixels with image frame format. A single bit per pixel places more pixels per character in the modulated plane format. In an embodiment where the drive controller can insert padding per block for the block sequential modulation plane and can accept input data of non-padded image frames, the image frame may not need to have block sequential padding.

Per-row padding

When Hsize is not Hmpix * Wsize and two rows of data in the same block are adjacent, the system that enables padding per row can insert padding between rows (at the end of the first row), making the data of the new row one word The boundary of the element starts. In some embodiments, a device may allow part of the data at the end of the first line to be packed into the first character of the next line, shifting all the data in the next line by a wrap amount. In this case, by not enabling padding per row, you can avoid padding per row. However, the padding at the end of the block data before the next block of some characters may still be needed, which is selected according to the padding of each block.

Per-set padding

Each set of padding is an option only available in raster sequential format. Only raster sequential formats (not block sequential) use line groups. Each group of padding is added to the end of each line group, ending with a one-character boundary. If each block or line padding is enabled, even if each set of padding is not enabled, the line group will still end at a character boundary.

Min-row-time padding

The minimum row time padding can add all or part of the padding characters at the end of a row or row group to allow the display device to have enough time to meet row strobe timing. If enabled, use a parameter to define how many word-times are needed to write one or more lines. Multiple lines can be written at the same time using the interleaved writing method. The amount of interleaving depends on the number of rows to be written simultaneously. A different value can be defined for each size of the available multi-line write options. For example, the minimum line character of one line at a time (for example, minimum line clock = MRC) can be 6 (MRCx1=6). The MRC for two rows at a time can be 6 (MRCx2=6); the MRC for four rows at a time can be 7 (MRCx4=7); the MRC for eight rows at a time can be 9 (MRCx8=9). First fill the data characters to the entire character boundary selected by other padding modes, and then subtract the words with valid data for each line or line group from the total number of characters (word-times) required for each line or line group The total number of yuan to calculate the amount of entire character padding required.

If some rows are written directly in block order, this can be applied to each row of the block data (depending on the megapixel size of this block, this may involve a single row write or multiple simultaneous writes).

For the raster sequence mode in which the data is grouped in line groups, the minimum line time padding is added only after the data in the line group ends. Depending on the number of line time segments required by the line group, different total time amounts may be required. The number of line time segments is determined by the crossing conditions of the blocks. If there is only a single group of unique line data (for example, when the entire line group is covered by a giant pixel row), only one line time is required. If there are two or more sets of unique row data, the row time of each group is added to obtain the total time required for the line group. When only one line group is written in one line time (for example, the line group is only covered by one line time), all lines can be written at the same time, and then the corresponding minimum line time can be used to calculate the required filling amount. For example, if four lines are written at the same time and there are four giant pixel characters given MRCx4=7, then three padding characters can be inserted/expected. If the line group intersects with the block Z0, all rows are written individually, LSS=4, MRCx1=6 and there are 22 characters to transmit the line group data, two padding characters are needed to make a total of 6*4=24 Characters/clock.

In all cases where this modulation plane protocol is applied, when the two formats use the same mode and sequence, necessary pixel padding is inserted into the gaze point image frame to facilitate direct conversion to the modulation plane format. The first step is to calculate the padding needed to modulate the flat characters. Then, in this image frame, a padding pixel is added for each padding in the modulation plane.

In some embodiments, the horizontal size (Hsizes) is used, where the horizontal size (Hsizes) is the product of the character size (Wsize) of the corresponding block and the size of the giant pixel, which can reduce the size of the block data in most cases. The required fill. For example, if Wsize=128, Z2Hsize=2048, Z1Hsize=1024 and Z2Hmpix=4, the number of Z2 characters outside block 1 is: (2048-1024)/4/128 = 2.0; however, if block 2 Smaller: (1920-1024)/4/128 = 1.75; or if block 1 is small (if the gaze is close to the edge): (2048-896)/4/128 = 2.25. This can also lead to padding in block 1; if the block size does not decrease when gazing close to the edge, the padding may be reduced overall (fewer characters).

Line-Set-Size (LSS)

The number of display lines written in a group (raster sequential mode). This also indicates the maximum number of lines that can be simultaneously written to a block/megapixel line of the same size as the LSS. Due to the physical circuit layout, the display device may have a fixed LSS, which may require the system and the driver to use this LSS or smaller. The value of LSS can also define a matching horizontal expansion capability. When this is not the case, another parameter may be used. Therefore, if the display can support the maximum LSS=4 (max Vmpix=4), then the display can also support the maximum Hmpix=4.

As a non-limiting example, if there is only a single block, the size of the line group can be one (LSS=1), which means that there is no reduction in fixation points; block 0 fills the display, and each line has Hres (horizontal resolution Degree) pixels and Vres (vertical resolution) number of pixel rows. In other words, Z0Hsize=Hres; Z0Vsize=Vres. This is the normal display mode at full resolution; each line has the same size: Hres bits.

As another non-limiting example, if there are only two blocks (for example, block 0 and block 1), where the vertical and horizontal macro pixel sizes of block 1 are 2^1=2 and LSS=2, then a line group Overwrite two display lines. Z1Hsize=Hres; Z1Vsize=Vres; Z1Hbits = Z1Hsize/Z1Hmpix = Hres/2. Some line groups will only be in block 1: One line of Z1Hbits bits corresponds to two rows of display pixels. It will need x=Z1Hbits/Wsize characters to transmit data for this line group. If this is a non-integer character and each block padding is enabled (or because there is only a single block in this line group, each set of padding is enabled), then it is rounded up. The last character in this group will have unused/filled data. In addition, when the minimum line time padding is enabled, if x> n, padding characters are added to satisfy n, where n = the minimum number of clocks for writing a single line. For example, this line group is only one giant pixel row.

The rest of the line groups will traverse block Z0 and block Z1. In this case, the block 1 character will be sent first, and then the two lines of block Z0 data will be sent. If there is a block Z1 data on each side of the block Z0 data in the display, the block Z1 data is packed together into the block Z1 character group. The display will separate them using the offset parameter from the header. The total characters of this line group are x0 (block Z0 characters) + x1 (block Z1 characters) + xp (filling characters); x0+x1+xp >= 2n.

As another non-limiting example, if there are three blocks and LSS=4, the line group covers four display lines. In addition, Z2Hsize=Hres, Z2Hmpix=4, Z1Hmpix=2. Some line groups will only be in zone Z2 (outside zone Z1): Hres/4-bit lines correspond to Z2Vmpix=4 lines of display pixels. Will need x2=Hres/4/Wsize characters to transmit the data of this line group. If this is a non-integer character and the corresponding padding is enabled, it will be rounded up. The last character of the group will have unused/filled data in it. In addition, when the minimum line time padding is enabled, if x2> n, padding characters are added to satisfy n.

If the line group spans all three blocks, the block Z2 data is sent first (to simplify the writing of the display line buffer), and the block Z1 and block Z0 data are based on just-in-time (JIT) In order and in order. Each character only contains data from the same block; there may be unused bits in the last character of the block group. Depending on the block size, it may not be necessary to pad characters to meet the minimum line time. Since each row in block 0 has unique data, each row is written once at a time. The total characters of the line group: x = x0+x1+x2+xp, where x>=4n. Because part of the line in blocks Z1 and Z0 is not included, the number of characters in block 2 data is less than the number in the previous example. Therefore, x2 = roundup((Z2Hsize-Z1Hsize)/Z2Hmpix/Wsize).

In some embodiments, it is also possible to use some non-standard formats, where low-res resolution (low-res) is expected to cover the entire display. It may be defined by multiple blocks, but the higher resolution block size is zero. It is also possible to define a format with only one block (block 0), but define the size of its giant pixels to be greater than 1. In this case, the LSS is also set equal to the size of this larger giant pixel to enable simultaneous line writing. The display device will have to be compatible with this format and system configuration.

Min Row Clocks

The display device has timing requirements to write a row into its array. At a given clock speed, this can be converted to the minimum number of clocks ("n"). When writing high-res data only for the overall display, there are many clocks/times written to each line. However, when transmitting a single row of giant pixels in a line group in a low-resolution block, the amount of data is much smaller and may be shorter than the time required to write a row into the array. For example, in the case of three blocks and one line group in block Z2, only one quarter of the clock/character is provided, which may be too short for the line write timing. By defining the required minimum number of clocks, the transmitter can fill in extra characters at the end of the online group data to meet the required timing. For the block Z2 with only the line group with LSS=4, only 1n (or 1*MRCx4) line time is required; therefore, all four lines can be written at the same time. For the block Z1 and Z2 line group, 2n (or 2*MRCx2) line time is required; thus these lines are written in pairs. Corresponding to the line group spanning all three blocks, 4n (or 4*MRCx1) line time is required; thus, these lines are written separately.

In some embodiments, the method described here does not use data enable or wait signals for dynamic flow control; this is a known and predictable timing requirement and can be solved by filling in the transmitted data.

Data Formats

FIG. 8A shows the data format of the gaze point image in the host memory according to some embodiments. The gaze point image in the host memory may include, for example, a full-color image of three blocks of giant pixels. As shown in the figure, the length of each row and the number of rows in each group depend on the block size and offset. In an embodiment of the present invention, even in a block, the system and/or method according to the present invention may have shorter rows, corresponding to areas with internal block cuts; where the zones on either side of the cuts The data on both sides of the block are connected together to save space and bandwidth/time.

FIG. 8B shows a data format of an image frame or a modulation plane according to some embodiments, wherein the data is sent sequentially through blocks with pads. It should be noted that, as shown in the figure, when using block end padding, the giant pixel data will end on the display character boundary. However, according to an embodiment of the present invention, the macro pixel data may not end on the video line boundary.

The video format of the standard image transmission interface can use the general line length, which does not match the block line length. The megapixel row is packed and wrapped into the valid data of the video row. In an embodiment of the present invention, there is a blanking time (blanking time) between each video interface line.

The flat data format is usually continuous data. The flat data format can be very similar to the image data format. They are only 1 bit per giant pixel instead of n bits per giant pixel. There is no horizontal synchronization (H-sync) or horizontal blanking (H-blanking).

The data format shown in FIGS. 8B, 8C, 8D, and 8E may be a full-color giant pixel format. For color-sequential formats, the data sequence after the header is repeated three times. For color-sequential data, the filling amount of the character boundary may be different.

FIG. 8C shows a data format of an image frame or a modulation plane according to some embodiments, wherein the data is sent sequentially through blocks with line padding. It should be noted that the line of block Z0 may need to be filled to end on the boundary of the display character, but usually the horizontal size (H-size) of block Z0 is on the character boundary.

FIG. 8D is a data format of an image frame or a modulation plane according to some embodiments, wherein the data format is determined by the raster sequence and the block group sequence with each block and line group line timing padding. send. It should be noted that according to the vertical offset of the block, the first line group usually only contains the data of the outer block, as shown in the figure. But it can have multiple blocks of data.

The horizontal size (H-size) of the display will be wider than any giant pixel row. The megapixel rows are filled to the character boundaries, and then packed together in the data stream. It should be noted that the block Z0 line may need to be filled to end on the display character boundary; but usually the horizontal size (H-size) of the block Z0 is on the character boundary.

FIG. 8E is a data format of an image frame or a modulation plane according to some embodiments. The data format is based on a raster sequence and time (Just-In -Time, JIT) are sent in sequence. The giant pixel data is packed into line groups and filled according to the character boundary and timing requirements, and then packed together in the data stream. It should be noted that according to the vertical offset of the block, the first line group usually only contains the data of the outer block, as shown in the figure. However, in some embodiments, there may be multiple blocks of data.

Multi-Drivers Per Array Column

For the multi-driver configuration, FIG. 9A shows the physical layout of the column multi-driver configuration according to some embodiments; the modulation plane is particularly compact to comply with the writing timing and line group of the pixel array with small data (for example, only across the highest macro). Pixel ratio block data) may require additional time to complete multi-line write operations. In some embodiments, multiple column drivers can be used to write multiple row line groups at the same time instead of adding padding pixels/bits to meet timing. The configurations of one, two, and four drives per column are shown. Various arrangements of multiple drives are shown, where 1, 2, 4, or 8 adjacent rows share the same drive; those skilled in the art should understand that these options are extended to larger groups.

FIG. 9B shows the number of row write times (number of row write times) required for the multi-drive arrangement of FIG. 9A for specific line group conditions and simultaneous grouping according to some embodiments. In the design, there is a trade-off between using multiple column drivers instead of wiring space and row write times. And there is a trade-off between another larger adjacent drive arrangement and additional buffering to write more line groups at the same time. Some systems or embodiments can also provide single pixel manipulation (vertical and horizontal) of the gaze point image, which results in misalignment of the line group with the adjacent driver arrangement, further aggravating the advantages of multiple drivers and requiring larger groups, Thus, compared with the case of a single drive per column, time saving can be achieved.

In order to overcome the minimum line time filling associated with large megapixels, each display column of the display device has multiple column drivers. Each driver can be divided into different groups of adjacent rows so that multiple different rows with different data can be written simultaneously; this allows the minimum row timing to be applied to multiple row groups. The data from the first row/line group can be buffered for use with the second row/line group, and so on. According to the present invention, by using the following parameters, these levels can be defined and included in the header, or with the system, method and/or agreement with the data sent to the display, the parameters are as follows:

1to1: This means that each column uses one drive.

2ps: This means that each column uses two drivers, and each line group uses one: one connects all the even-numbered line group rows, and the other connects all the odd-numbered line group rows. This allows the data/clock from the two-wire group to be applied to the timing required for the rows of the one-wire group. This is assuming that the two line groups span the same number of blocks (V step size should be 2*LSS). Any padding still needed is applied after this second line set.

2ph: This means that each column uses two drivers, and each half of the line group uses one: one is connected to the upper half of the line group row, and the other is connected to the lower half of the line group row. This allows the line group to span the two largest blocks, which usually requires simultaneous two row times (row times) writes; use the clock for the entire line group to comply with MRC. Compared to the full line group mapping, this option may be less inclined.

4ph: This means that each column uses four drivers, and each half of the line group uses one: one line connects all the upper half of the even line group, the next connects the lower half of the even line group, and the next connects all the odd lines The row in the upper half of the line group, and the last one connected to the row in the lower half of the odd-numbered line group. This provides more time for the largest block and the two largest blocks. Similarly, the vertical step size (V-step-size) should be 2*LSS.

4ps: This means that each column uses four drivers, and each line group uses one: one connects the first of the four line groups, the next connects the second of the four line groups, and the next connects the first of the four line groups. Three, the last one connects to the fourth of the four wire groups. This allows the data time from the four wire groups to be applied to the timing required for the one wire group. (V-step-size should be 4*LSS).

As a non-limiting example, if the display uses a 2ps multidrive (header selection MD=2ps) and LSS=4, then for the minimum line time padding, it is like treating two line groups as one. However, the display buffers data to write two line groups simultaneously. Therefore, n only applies to the number of clocks of the line group combined in block 2; among them, if in blocks 1 and 2, the number of clocks is 2n; and if the line set crosses block 0, then The number of pulses is 4n. It should be noted that this also limits the size and offset of the block in the vertical direction to a multiple of eight display lines. In this way, both groups of the four rows have the same block across the condition.

Macropixel ratios, pixels per degree, and cones per pixel (Macropixel ratios, Pixels-per-Degree and Cones-per-Pixel)

Figure 10 shows the diameter, field width, and cone density of each region of the human eye relative to the gaze point obtained from public sources. This data is used as an example of optical design considerations for the gaze point display system.

The most direct way to analyze the human visual response that matches the characteristics of the display hardware is to use binary multiples in the pixel resolution (reasonable values of integer step size and other options will be discussed later). For example, let the high-resolution giant pixel in the center be a 1x1 display pixel, and the next step to reduce the resolution is a giant pixel whose size is equivalent to a 2x2 high-resolution pixel, the next is 4x4, and then 8x8. So on and so forth. However, since the maximum ratio between the center and peripheral cone density is 10-14:1 (see Figure 10 for details), there is no binary multiple above 8x8 to maintain human visual acuity. According to an embodiment of the present invention, the horizontal macro pixel ratio and the vertical macro image ratio of the block are the same, for example, resulting in a square array of display pixels for each macro pixel; however, the size of rectangular macro pixels can be used. In an embodiment of the present invention, the ratio of each macro pixel may be an integer, which enables the display device hardware to copy (or write in) the bit or value of each macro pixel in the horizontal and/or vertical direction. ) To the bit or value of the corresponding display pixel.

Imaging applications usually choose the resolution of the central area, which matches the average resolution of an area, where the area is larger than the foveola or foveola avascular area (FAZ); therefore, this application does not use the highest Fovea density. The display usually does not include the entire periphery of the user's field of view (FOV); therefore, the lowest sensitivity of human vision at the periphery is not used. The resolution of each of the lower resolution blocks is selected based on the highest sensitivity part of the block on the inner boundary of the block. Or in other words, the boundary of the visual acuity inner block is selected based on the selected resolution down to the block. This means that the lowest system resolution will be based on the sensitivity of the boundary between the two lowest resolution blocks, which is an angle much smaller than the total field of view (FOV). Therefore, considering all these three factors, the ratio of the highest to the lowest resolution can be only 4:1.

The number of cones of each display pixel in the central high-resolution block is another method for selecting the optical system to make the resolution suitable for the field of view. The industry believes that a "retinal display" has 60 pixels per degree of VF in the central area, which is approximately equivalent to 5-8 viewing cones per pixel (among them, the small concave area: on a plane, 500 cones/mm* .35 mm/degree/60 pixels/degree = 2.9 cones/pixel; 2.92 = in an area, 8.5 cones/pixel. Foveal avascular area (FAZ): a plane Above, 400 frustum/mm * .5 mm/1.5 degrees/ 60 pixels/degree = 2.22 frustum/pixel; 2.222 = 4.9 frustum/pixel in one area). The fixation point display system and method do not select the frustum pixel threshold or sensitivity; it is controlled by the application. The fixation point display system and method only associate the resolution ratio of the center block with the lower resolution block.

As mentioned above, using the quadratic step of resolution, the peripheral vision requirement is less than a quarter of the linear resolution of the central area. Therefore, using 4x4 megapixels (1 megapixel represents 16 display pixels, which is equivalent to a 1/16 reduction in area resolution) is a corresponding option. Therefore, the resolution of the external fixation area can be defined as 4x4. There is only one binary step between 4x4 and 1x1, which is 2x2. Therefore, three blocks are defined, one of which coincides with each binary resolution step. Other numbers of blocks and resolution steps can be supported, but these three are typical for the fixation point display system and method.

Sizes of Zones or Transition Between Zones (Sizes of Zones or Transition Between Zones)

The next consideration may be the conversion between blocks and the total reduction in bandwidth. In an embodiment of the present invention, the position of the conversion between the blocks depends on the optical system parameters and the requirements of the host processor. This gaze point display system and method are agnostic of the size of each block (as discussed later, there is a small limit on the overlap area and the step size in the area). It is illuminating to observe the shape of the gaze point area to understand how much the host will reduce the resolution of the control. To analyze the resolution from the perspective of the user's field of view, each conversion point can be defined from the angle of the gaze center VF. Perform the conversion from low resolution (4x4 megapixels) to medium resolution (2x2 megapixels) at the boundary between Perifovea and Mid Peripheral (because its cone density is the center The linear quarter of the area), which is approximately 9° from the center. The system can add some tolerance to allow eye tracking accuracy and delay. If the tolerance is 5°, this conversion can be pushed to ±14°. Similarly, the boundary between Fovea Centralis and Parafovea can be converted from medium resolution (2x2 megapixels) to full resolution (1x1 pixels) (because its cone density is the center Half of the area), which is approximately ±3°; under the same tracking tolerance, the block conversion can be approximately ±8°. Some systems with extremely wide Field of View (FOV) can even accept a third conversion from one-quarter resolution to one-eighth resolution.

All conversion considerations have almost no effect on the fixation point display system and method. The system designer will have to determine the worst-case block size to calculate the maximum bandwidth requirement. As shown later, a typical configuration with three blocks has a field of view (FOV) of 60 degrees for each eye, reducing the bandwidth to one-sixth; a wider FOV and tighter tracking tolerance will Further reduce it.

Digital Display Device

Digital display devices usually use a memory array structure to control the pixels of the display; each column in the display has a column data driver and each row has a writing function. Therefore, all rows of pixels can be written at once. Therefore, data from different blocks overlapping with the same row can be combined together before the row can be written. Some display devices may have block enablement to allow only part of a row to be written at a time, which may allow only part of the line matching the gaze area to be written, which will allow direct block sequential writing. However, it will take extra time to write some rows multiple times to address all the pixels in the row. The block activation boundary may impose unbearable restrictions on the gaze zone boundary in order to align it, which makes the block larger, thus limiting the reduction of data bandwidth. Very low-resolution blocks may not be able to be written into each row group in a short time to receive its reduced data group, so it is necessary to fill/restrict the lowest block to reduce its efficiency. For these reasons, the activation of blocks used for partial row writing will have limited benefits. Some devices may divide these rows and have multiple activations; this concept in the gaze point display system and method still applies.

In order to transmit block definitions and other metadata (meta data), a header may be used at the beginning of the plane to define the total image or plane size, the size and location of each area of interest, and the parameters that control the order and data packaging. Metadata can also be transmitted as side-band data other than normal video data (for example, command data packets in the vertical blanking interval). The header method at the beginning of the video data can be packaged as a parameter per pixel, or covered as a single bit with a parameter per pixel (like the data are all zero or all one), so as to be easily transferred to the modulation Flat format. For example, in a protocol configuration for a flat interface, there are three blocks and the data is grouped into four sets of output lines, where each data character contains data of one resolution from only one block (the character size is determined by Defined by the physical interface of the display device and used by the host to format the packaged data). Depending on whether the lines are only in block 3 or they span multiple blocks, the number of characters required for each four-line group will be different. Therefore, as the image is transferred from top to bottom, the transmitted data size of each line group will be larger or smaller; there is no constant line size. In the same example, the display device can include the ability to simultaneously write up to four rows of pixels from a small segment of input pixels; it can also save and expand low-resolution megapixels, and mix them with high-resolution pixels into multiple Row, and then self-time the writing of each row.

作為範例，總圖像/顯示區域可分割為如下所示的三個區塊：

As an example, the total image/display area can be divided into three blocks as shown below:

There are many types of display devices used in this fixation point display system and method. Some monitors are used for monochrome, some monitors are full color at the same time, and some monitors are color sequential. Some displays inherently express the full intensity range as fractional steady-state intensity (such as analog displays or self-adjusting digital displays); other displays are digital in nature and only drive to a binary stable level, which requires Update the modulation plane frequently to modulate this intensity. The display usually has a frame buffer memory for storing image data, and then uses the data from the buffer to drive and make the display emit light; usually there are two buffers for alternate operations (in one frame, write new frame data into one frame Buffer, and read another buffer to display the previous frame data; the next frame they exchange read and write operations); but if the transmission data format/protocol/bandwidth supports this display sequence, the optimized design benefits from a single buffer . These systems with the highest requirements for transmission and writing timing are color sequential, binary displays with a single frame buffer. The rest of the description will focus on this configuration, but the gaze point concept in the gaze point display system and method can also be applied to other display configurations.

Please refer now to FIG. 7, which is a timing diagram of transmission illumination according to some embodiments, showing color sequence images and planes of the buffer type. As shown in the figure, multiple timing diagrams are drawn for the various protocol options of these configurations. Specifically, the timing diagrams of image transmission, plane transmission, display writing, and lighting for different configurations of frame buffer, image data number sequence, and display type are shown. All of these represent displays used in color sequential lighting. In particular, the display and driver hardware configuration can have one, two (or more) frame buffer memories, or no frame buffer memory. Most display drivers use a dual ping-pong frame buffer architecture. Color display architectures are usually three-path (3-path) lighting and optics or color sequence; color sequence is a more economical and timing-challenging method; although the fixation point agreement improvement can also be applied to monochrome or three-path systems, these diagrams And the description still focuses on it. The sequence is usually the same: at the beginning of the vertical synchronization (Sync), the image must first be transferred/transmitted from the host/source to the display driver, processed as needed and stored in the frame buffer. Color sequence systems usually have more than three color sub-frames (CSF) per input image frame to provide multiple sets of main pulses to improve image quality. After receiving the image, the frame buffer can be read to form the CSF data and transmitted through the flat (or grayscale) transmission interface; then the data is stored/used in the native display elements to represent the image Like the CSF part; repeat this operation for each CSF.

In a system with a frame buffer, this buffer should not be read while writing data; this will cause new data to be mixed with old data (or damaged); it must wait for the image writing to complete. For full-color pixel images, the entire image must be completed before any CSF reading starts (as shown in the first two figures). For modulated flat panel displays, illumination usually starts shortly after the modulated plane starts (the corresponding frame buffer data is usually read multiple times; once for each modulated plane). For grayscale displays, light emission will not be activated when the display is updated with new CSF data. It is generally desirable to use a lighting system with the highest duty factor (minimize the time without active lighting). Therefore, these full-color single frame buffer systems try to minimize the image transmission time and grayscale transmission time; this is where the gaze point data protocol reduces the bandwidth the most. In addition, the modulation plane transmission system is also limited by the time of transmitting each plane, which limits how many modulation pulses can be accommodated in a light-emitting cycle, or limits the gamma level that can be achieved. Pack some of the modulated pulses more tightly than others. By significantly reducing the data (size) of the transmitted plane, the fixation point plane transmission solves this bottleneck, so the modulation pulse can be shorter.

The bandwidth of a flat or grayscale transmission interface is usually significantly higher than that of an image transmission interface (because it is usually a short and wide chip-to-chip interface or an internal chip interface). The host usually uses most of the frame time to transmit images; for a system with dual-frame buffers that are used in an alternating manner (switching write and read buffers at each Vsync), a small part of this frame is more There is no benefit in writing images quickly. However, motion sensitive applications require faster response and will benefit from faster enabling of buffered reading. If the host can send image data in a color sequential format, only one frame buffer is needed; one color is written into the frame buffer, and the other colors are read and emit light. This provides high duty cycle lighting and low latency (the time from the start of transmission to the start of matched lighting). As long as there is a set of main lighting periods (3 CSF’s), almost the entire frame time can be used to transmit images. The higher CSF's group requires that the transmission time be reduced to a lesser part of the frame time, and still be consistent with a frame buffer (for example, image writing must meet the reading time of 4 CSF's); the reduced data of the fixation point agreement can be Enable these functions without significantly increasing the bandwidth of the physical interface.

If the display device supports direct grayscale writing and only uses three CSFs, frame buffering is not required (the last image); this does require a pause in the image transmission between color sub-frames to allow light emission. This light-emitting duty cycle depends on the image transmission time and frame rate; using fixation point transmission and fixation point writing, this may still be relatively high. This is mainly suitable for systems with high frame rates or insensitive to movement (only 3 CSFs can be tolerated).

The digital display can receive modulated plane data (1 bit per pixel), and the modulated plane data is repeated multiple times per frame or sub-frame in various pulse patterns to integrate the desired gray scale. This protocol and method are mainly aimed at modulating the plane interface level. Although it can also be applied to a display that accepts the grayscale data of each pixel and selects its own pulse pattern, it is still limited by the internal array structure.

In order to integrate the pulses so that the human visual response treats each pixel as a constant gray scale or color scale, each pixel of the display (for example, through each modulation plane) is fastened at a frequency such as 10 kHz to 100 kHz To update. Use a column driver for each pixel of the row (shared among all/many rows), write the pixel array one or more rows at a time, and use a unique row strobe (unique row strobe) for each row. The key timing constraints for writing in the display are the time to write a row and how many rows can be written at the same time. This method defines high and low resolution blocks, which can be easily combined and cover the writing of the row structure, so that each modulation plane updates the overall effective area.

In the interface of the modulation plane, since the pixel has only 1 bit depth, there is no opportunity for planting. Optionally, averaging multiple pixels will eliminate the benefit of spatial jitter, which is already encoded in the data. Even if each pixel has available multi-bit metadata, creating special pixel data in the extended low-resolution area will avoid saving time for writing multiple rows with the same column of driver data at the same time. Most monitors will not modify the input data, so it is limited to copying in low-resolution areas.

Amplitude v. Phase Mode

According to the gaze point display system and method of the present invention, if the display device is used in the amplitude mode to provide a visible image, each pixel on the display corresponds to a pixel (or a small part of the phase) in the image during viewing or projection. Adjacent pixels), and then the fixation technology enabled by the present invention can be used to achieve high spatial frequency and/or time frequency update in the fixation area of the innermost block, and to achieve low resolution/spatial frequency in the outer block High time-frequency update of the area, where the gaze area of the innermost block corresponds to the image area that the viewer is watching or is considered to be watching from the nearest distance, and the low resolution/spatial frequency area in the outer block corresponds to Peripheral vision of the viewer. In such an amplitude mode display, there is essentially a one-to-one relationship between multiple display pixels/megapixels and image pixels. Therefore, there is a similar inter-area mapping in the display area, which can help to receive more information for matching use The high sensitivity area of the visual system.

However, in a phase mode display, many display pixels (in some implementations, all display pixels) contribute to each image pixel (in some implementations, all image pixels). In this case, the area of high sensitivity or the area of the observer's visual system can be mapped to a substantially larger (in pixel count) area on the display. In such a system, many different algorithms or methods can be used to generate various patterns, such as computer generated holograms (CGHs), interferograms, holograms, interference patterns, and phase maps. The image or the version obtained therefrom is displayed on the display device as a distribution of phase value, amplitude value or a complex combination of phase value and amplitude value, so that light is diffracted or scattered from the displayed pattern or propagated in other ways The second pattern is formed at a certain actual or optical distance from the display device, which corresponds to the desired image or can be obtained through other optical devices. In at least some algorithms or methods, part or all of the lower spatial frequency content of the desired image is encoded or included on the displayed image in a region, where the region substantially or completely surrounds the desired portion Or all higher spatial frequencies are encoded or contained in a larger area of the display image. In such a case, the system and method and the gaze point display method of the present invention can be advantageously used to provide higher spatial frequency and/or timing frequency updates to the outermost block or the middle block.

In addition, there are also amplitude mode applications, in which the area of the display that can most advantageously receive higher spatial frequency and/or time frequency updates may be the outer or outermost area instead of the innermost or inner area. For example, for low-brightness scenes (also called "night vision") viewed by observers in scotopic or mesopic rather than photopic mode, the maximum visual sensitivity (for motion, color, and amplitude) Or other visual parameters) can be outside or around the gaze area.

The following are several embodiments of the present invention. However, other embodiments are described above. Therefore, the following embodiments are not intended to limit the embodiments of the present invention.

A fixation point (gaze point) light modulation system, comprising: a processor coupled to receive an input image data and a fixation point area definition data to generate a fixation point image frame, the fixation point image frame There is a header packet data indicating a first block with a first resolution and a second block with a second resolution, wherein the second resolution is smaller than the first resolution Degree, and wherein at least one of the first block and the second block is compressed based on a giant pixel ratio (macro pixel ratio); a drive controller circuit coupled to receive the gaze point from the processor Image frames, and the processor generates a plurality of modulation planes based on at least part of the bit plane data of the gaze point; and a modulation device having a plurality of pixels including at least one giant pixel, and the modulation device is coupled to the The driving controller circuit receives the modulation planes, and each of the modulation planes is expanded based on the header packet data.

A gaze point light modulation system, in which for a second block with a second resolution, a single bit in the modulation plane represents a giant pixel of the modulation device, and this single bit is copied to the giant pixel The subset of display pixels defined by the ratio.

A gaze point light modulation system, wherein the modulation device is a display.

A gaze point light modulation system, in which the modulation device is a liquid crystal on silicon (LCOS) display.

A gaze point light modulation system, wherein the modulation device includes a decoding logic module coupled to receive a modulation plane.

A gaze point light modulation system, wherein the modulation device further includes raster logic, and wherein the decoding logic module parses the header packet data and the raster logic.

A gaze point light modulation system in which the decoding logic generates a data set based on the ratio of the header packet data to the megapixels in the modulation plane.

A gaze point light modulation system, further comprising: a tracking logic module coupled to the processor, wherein the tracking logic module senses retinal gaze data and head position data of the user, and generates gaze point block data corresponding to the sensed Retinal gaze data and head position data.

A gaze point light modulation system, wherein the processor further includes: a gaze point rendering module is coupled to the tracking logic module and receives the sensed retinal gaze data and head position data, and uses a gaze point rendering algorithm to determine each zone The size and location of the block.

A fixation point light modulation system, in which fixation point rendering module is based on total field-of-view data, optical system distortion data, and fovea acuity data ), tracking logic data tolerance (tolerance of tracking logic data), tracking logic data latency (latency of tracking logic data) and movement data rate (rate of motion data) at least one, using the fixation point rendering algorithm to determine each area The size and location of the block.

A gaze point light modulation system, wherein the header packet data includes: the resolution-sequence trigger bit for enabling a center detail mode and a peripheral detail mode, wherein when the center detail mode is activated, the gaze point image frame contains A plurality of concentric blocks, including a block with the highest resolution of the first and second blocks located at the center of the user's gaze point, and thus a zone adjacent to at least one of the first and second blocks The resolution of the block is lower than the full resolution of the other adjacent blocks by a predetermined value, and the resolution of one of the concentric blocks decreases in descending order from the direction away from the user's gaze point; and When the detail mode is activated, the gaze point image frame includes a plurality of concentric blocks with the highest resolution on the periphery of the concentric blocks, so that the resolution of the second block is lower than that of the first block. The resolution of the block is a predetermined value, and the resolution of each concentric inner block decreases in descending order; the transmission mode switch bit enables a raster sequential mode and a block sequential mode, and when the raster sequential mode is active When the data transmission includes multiple line groups, the line groups represent data rows from the concentric blocks, corresponding to the display sequence of an original image, and when the block sequential mode is active, the data transmission includes Before the data transmission of an adjacent block, the complete transmission of each of the concentric blocks; the block number field defines the number of the concentric blocks; the block size field defines each of the concentric blocks A horizontal and vertical size; a block offset section, which defines the horizontal and vertical offset of each of the concentric blocks; and multiple display parameters.

A gaze point light modulation system, wherein the display parameters include: a character length segment, defining a plurality of pixel bits corresponding to the timing cycle related to the drive controller circuit; and an x offset size segment, Define multiple pixel bits in the Least Significant Bit (LSB) of each horizontal offset; a line group size segment defines the maximum number of lines to be written at the same time; a line time segment defines the amount required to write a line A timing segment; and a dual-column drive mode indicator, capable of writing two rows at the same time.

A method for generating a fixation point image on a display screen includes: receiving input image data regarding the fixation point image and a plurality of fixation point area parameters and a plurality of protocol parameters; based on the input image data and these The gaze zone parameters generate rendered gaze point image data, and a gaze point image frame is generated based on the rendered gaze point image data and the protocol parameters, wherein the gaze point image frame includes header packet data, and The gaze point image frame is transmitted to at least one of the grayscale device and the modulation device.

A method in which the gaze point image frame identifies two or more concentric blocks with different resolutions, so that each of the concentric blocks is defined by a plurality of macro pixels and a corresponding macro pixel ratio.

A method, wherein at least one of the grayscale device and the modulation device includes decoding logic and raster logic.

A method, wherein at least one of a grayscale device and a modulation device includes a plurality of pixels, and further includes: based on the header packet data and the corresponding ratio of each of the macro pixels, using the raster logic to connect the grayscale device and At least some pixels of at least one of the modulation devices generate a fixation point image.

A method in which for a gaze area with a reduced resolution, based on the corresponding macro pixel ratio associated with each block, a single bit of the associated multiple macro pixels is copied to a subset of the display pixel array.

A method, wherein receiving image data includes: receiving tracking data based on at least one of retina and head gaze direction data and real-time user position/location data.

A method in which the gaze point image data is based on the input image data, gaze point area parameters and tracking data.

A method, wherein generating a rendered gaze point image includes: using a gaze point rendering 3D to 2D rendering technology or other gaze point rendering technology known in the industry to generate image megapixels, to use input image data, observer Gaze direction data, vantage point data, and multiple gaze point area definition parameters represent projected text or graphics in the gaze point image space.

A method, wherein generating a gaze point image frame includes: generating header packet data based on gaze point area parameters, selected protocol parameters, and capabilities of a display device; and encapsulating the rendered gaze point image with the header packet data According to the data, a gaze point image frame is generated.

A method, wherein transmitting the gaze point image frame includes: transmitting the gaze point image frame to a drive controller circuit; generating gaze point bit plane data; and based on an associated modulation format and the header packet data Convert this gaze point bit plane data into multiple modulation planes; and transmit the modulation planes to one or more modulation devices, wherein the one or more modulation devices have gaze point modulation plane grating logic , Which is coupled to a display circuit with a pixel array, or transmits the gaze point image frame to a grayscale display device.

A method, wherein generating a gaze point image on a display pixel array includes: parsing header packet data and gaze point data from a gaze point image frame or a gaze point modulation plane; translating a modulation plane of the gaze point data. ) Is the gaze point display data; and the binary value of the corresponding gaze point display data representing the line group is applied to each subset of the pixel array associated with each gaze point block; and the aforementioned translation and application are repeated until Each line group of the gaze point data is displayed.

A method in which the detection of the activation of the raster sequence mode and the block sequence mode includes: parsing the header packet data to detect the transmission mode trigger bit; and detecting whether the transmission mode trigger bit is set to enable the raster sequence mode and the block Sequence mode, and in response to the raster sequence mode not being detected, detects individual block data of the incoming data based on the number of blocks, horizontal block size, line group size, character size and x offset size.

A method in which writing in response to the enabled block sequential mode includes: in response to not detecting the raster sequential mode, writing individual block data of the incoming data into the corresponding block buffer {Z(n- 1), …, Z3, Z2, Z1, Z0}; in response to not detecting the raster sequence mode, waiting for a read delay for a predetermined time, and in response to the end of the read delay, switching corresponds to each individual block Buffered reading indicator.

A method, wherein identifying whether data from a corresponding block exists includes: parsing header packet data to detect block number, horizontal block size, line group size, line time, character size and x offset size, where the line The time is based on the number of clocks (clocks) required to write a row based on the display circuit; select one row of the data line group; detect the existence of a high-resolution block (Z0) of multiple concentric blocks based on the header packet data In this line; and in response to the existence of no high-resolution block detected, detect whether the next consecutive block (Z1, Z2, Z3, ... Z(n-1)) exists in the identified data The line group until a block is detected.

A method in which the expansion of the line group into the gaze point display data includes: parsing the header packet data to detect the number of blocks, the size of the horizontal block, the size of the line group, the line time, the character size and the x offset size, where the line The time is based on the number of clocks required to write a row based on the display circuit; based on the line group size and line time to select the line group of the identified fixation data; identify a left segment and A right segment; when an individual block {Z0, Z1, Z2, Z3... Z(n-1)} is detected, this left segment is written into a row of buffers, and its displacement x offset corresponds to the individual The block data of, where the left segment is written 2(r-1) times, so 2(r-1) represents the corresponding giant pixel ratio, where r = {n, n-1, n-2, n-3 , …1}, n is the number of blocks; add the horizontal block size and x offset size of each individual block data, and define the right index for each right segment of the individual block data; write the right segment Enter the row buffer, the row buffer is shifted by the indicator on the right side of the individual block data; the row buffer is stored in row to column, where k rows are written 2 (r-1) times, k = {1, 2 , 4, 8,… 2(n-1)}; and repeat the aforementioned selection, identification, writing, summing, writing and storing until each row of the identified line group is selected.

A non-transitory computer readable medium, including encoding for executing a method, the method comprising: receiving image input data; receiving tracking data based on the position of the user's retina in real time; generating a mark based on the image input data and tracking data Header packet data to define fixation point block information; Encapsulate image input data in header packet data to form a fixation point image frame; Transmit the fixation point image frame to one or more modulation devices, wherein Each modulation device has a pixel array; converts the gaze point image frame into multiple modulation planes; analyzes the header packet data and gaze point image data from each modulation plane; modulates the gaze point image data The plane is translated into gaze point display data; and the binary value representing the corresponding gaze point display data of the line group is applied to each subset of the pixel array associated with each gaze point block.

A computer readable medium, wherein the modulation plane for converting the image data of the gaze point includes: parsing the header packet data to detect the number of blocks, the size of the horizontal block, the line group size, the line time, the character size and the x offset The line time is based on the number of clocks required to write a line based on the display circuit; whether the transmission mode trigger bit is set to enable the raster sequence mode; in response to the raster sequence mode not being detected, based on Block number, horizontal block size, line group size, line time, character size, and x offset size to detect the corresponding block data of the incoming data; in response to the raster sequence mode not being detected, all The corresponding block data of the incoming data is written into the corresponding block buffer {Z3, Z2, Z1, Z0}; in response to not detecting the raster sequence mode, waiting for a preset time of read delay; in response to reading At the end of the delay, toggle (toggling) corresponds to the read pointer of each individual block buffer; in response to detecting the raster sequential mode and in response to the end of the read delay, identify the first line group of the data One line; detect whether a high-resolution block (Z0) of multiple concentric blocks exists in the identified data line group; in response to not detecting the existence of a high-resolution block, detect the next continuous zone Whether the block (Z1, Z2, Z3) exists in the identified data line group until a block is detected; in response to detecting a block, it is based on the number of blocks, horizontal block size, line group size, x The offset size, line time and character size are used to expand the data; the expanded data is stored in a line buffer based on the x offset size; when an individual block {Z0, Z1, Z2, Z3} is detected, the This row buffer is transferred to row to column, where k rows are written 2 (r-1) times, r = {n, n-1, n-2, n-3, …1}, k = {1 , 2, 4, 8, ... 2(n-1)}, n is the number of blocks; retrieve the next line group of the data; and repeat the aforementioned detection, expansion, storage, transmission and retrieval until each of the data The line group is retrieved.

A computer readable medium in which, in response to detecting a block, the expanded data based on the number of blocks, horizontal block size, line group size, line time, and character size include: Identify the left side of individual block data Segment and right segment; write the left segment into a row of buffer, the displacement x offset size corresponds to individual block data, where the left segment is written 2 (r-1) times; add each individual block data The horizontal block size and the x offset size are used to define the right index for the right segment of the individual block data; write the right segment into the line buffer, which is shifted by the right index of the individual block data; store this line buffer Go line by line; and repeat the aforementioned identification, writing, adding, writing and storing for each row of individual block data.

In the above description, many details are explained. However, it is obvious to those skilled in the art that the present invention can be implemented without these specific details. In some cases, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid obscuring the present invention.

It should be understood that the above description is intended to be illustrative rather than limiting. From reading and understanding the above description, many other embodiments will be apparent to those skilled in the art. Although the present invention has been described with reference to specific exemplary embodiments, it should be understood that the present invention is not limited to the described embodiments, but can be implemented with modifications and substitutions within the spirit and scope of the appended claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive. Therefore, the scope of the present invention should be determined with reference to the scope of the attached patent applications and the full scope of equivalents conferred by the scope of these patent applications.

This article discloses detailed illustrative implementations. However, for the purpose of describing the embodiments, the specific functional details disclosed herein are only representative. However, the embodiments may be embodied in many alternative forms, and should not be construed as being limited to the embodiments set forth herein.

It should be understood that although the terms first, second, etc. may be used herein to describe various steps or calculations, these steps or calculations should not be limited by these terms. These terms are only used to distinguish one step or calculation from another. For example, without departing from the scope of the present disclosure, the first calculation may be referred to as the second calculation, and similarly, the second step may be referred to as the first step. As used herein, the terms "and/or" and the "I" symbol include any and all combinations of one or more of the associated listed items. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly dictates otherwise. It will be further understood that when used herein, the terms "comprises", "comprising", "includes" and/or "including" designate the presence of said features, integers , Steps, operations, elements and/or, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof. Therefore, the terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting.

It should be further noted that in some alternative implementations, the mentioned functions/acts may occur out of the order indicated in the figures. For example, depending on the functions/actions involved, two figures shown in succession may actually be executed substantially simultaneously, or may sometimes be executed in reverse order. In consideration of the above embodiments, it should be understood that the embodiments can employ various computer-implemented operations involving data stored in a computer system. These operations are operations that require physical operations on physical quantities. Generally, although not required, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. In addition, the operations performed are often referred to in terms, such as generating, identifying, determining, or comparing. Any method of operation described herein that forms part of the embodiment is a useful method of machine operation. The embodiments also relate to equipment or devices for performing these operations. The device may be specially constructed for the required purpose, or the device may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written according to the teachings herein, or it can be more convenient to construct more specialized devices to perform required operations.

The entity operable by modules, applications, layers, agents, or other methods can be implemented as hardware, firmware, or a processor that executes software, or a combination thereof. It should be understood that in the case of software-based embodiments disclosed herein, the software may be embodied in a physical machine such as a controller. For example, the controller may include a first module and a second module. The controller may be configured to perform various actions such as methods, applications, layers or agents.

The embodiments can also be implemented as computer readable codes on a non-transitory computer readable medium. The computer-readable medium is any data storage device that can store data, which can then be read by a computer system. Examples of computer-readable media include hard disk drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, flash memory devices, and Other optical and non-optical data storage devices. The computer-readable medium may also be distributed on computer systems on a network, so as to store and execute computer-readable codes in a distributed manner. Various computer system configurations including handheld devices, tablet computers, microprocessor systems, microprocessor-based or programmable consumer electronic products, minicomputers, main frame computers, etc. can be used to implement the embodiments described herein. The embodiments can also be implemented in a distributed computing environment. In the distributed computing environment, tasks are performed by remote processing devices connected through a wired or wireless network.

Although this method is described in a specific order, it should be understood that other methods can be performed between the described methods, and the described methods can be adjusted so that they occur at slightly different times. Or the described operation values may be distributed in a system that allows processing operations to occur at various intervals associated with processing.

In various embodiments, one or more parts of the methods and mechanisms described herein may form part of a cloud computing environment. In such an embodiment, resources can be provided as services on the Internet according to one or more models. Such models can include infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). In IaaS, the computer infrastructure is delivered as a service. In this case, the computing device is usually owned and operated by the service provider. In the PaaS model, the software tools and basic equipment used by developers to develop software solutions can be provided as services and hosted by service providers. SaaS usually includes software licensed by the service provider as an on-demand service. Service providers can host the software or deploy the software to customers within a given period of time. Various combinations of the above models are possible and predictable.

Various units, circuits, or other components may be described or requested as "configured to" perform one or more tasks. In such a context, the term “configured to” is used to indicate a structure by indicating that the unit/circuit/component includes a structure (for example, a circuit) that performs one or more tasks during operation. In this way, even if the specified unit/circuit/component is not currently working (for example, not turned on), it can be said that the unit/circuit/component is configured to perform the task. The units/circuits/components used in the “configured as” language include hardware; for example, circuits, memory storing executable program instructions to realize runnability, and the like. Explicitly "configuring" a unit/circuit/component to perform one or more tasks is obviously not to trigger the U.S. Patent Law (35U.S.C 112, sixth paragraph) for that unit/circuit/component. In addition, “configured to” may include a general structure (for example, general circuit) manipulated by software and/or firmware (for example, an FPGA or a general-purpose processor that executes the software) to operate in a manner capable of performing the following operations. "Configuring to" may also include adapting a manufacturing process (such as a semiconductor manufacturing facility) to manufacture a device (such as an integrated circuit) suitable for accomplishing or performing one or more tasks.

For the purpose of explanation, the foregoing description has been described herein with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. In view of the above teachings, many modifications and changes can be made. The embodiments are selected and described in order to best explain the principles of the embodiments and their practical applications, so that those skilled in the art can best utilize the embodiments and various modifications to suit the specific intended use. Therefore, the present embodiment should be considered as illustrative rather than restrictive, and the present invention is not limited to the details shown here, but can be modified within the scope of the attached patent application and its equivalents.

100: Gaze point electromagnetic radiation modulation system 103: Gaze point image frame 105: input data source 107: Tracking Logic 110: processor circuit 111: block definition module 112: Gaze point rendering module 114: Gaze point image 115: image protocol encoding 120: drive controller circuit 122: memory 123: Image protocol decoding 124: Conversion unit (image transposition plane) 125: plane agreement coding 126: Conversion unit (bit plane to modulation plane) 127: Modulation Plane 130: Modulation device circuit 132: Memory 133: flat protocol decoding 140: display circuit 142: Control Unit 144: pixel array circuit 146: Display 150: Raster logic 152: Wire group collection circuit 154: write logic directly 156: line buffer 158: Row to Column 162: Grayscale device circuit 163: Image protocol decoding 164: Memory 165: display 170: display circuit 172: Control Unit 174: pixel array circuit 180: Raster logic 182: Wire group collection circuit 184: Write logic directly 186: line buffer 188: Row to Column 490: Schematic 494: line group fragment 600: computing device 602: Central Processing Unit 604: Memory 606: Bus 607: Video Drive 608: Mass storage device 610: input/output device 612: display Z0~Zn: block

By referring to the following description in conjunction with the accompanying drawings, the described embodiments and their advantages can be fully understood. These drawings do not limit any changes in the form and details of the described embodiments by those skilled in the art without departing from the spirit and scope of the described embodiments.

1A is a system schematic diagram of a gaze point electromagnetic radiation modulation system according to some embodiments. The system has a processor circuit, a display driving circuit, and a gaze point modulation planar device circuit.

FIG. 1B is a system schematic diagram of a fixation point electromagnetic radiation modulation system according to some embodiments. The system has a processor circuit and a fixation point grayscale device circuit.

FIG. 2A is a flowchart of a method for generating a fixation point image frame by the processor circuit 110 of FIG. 1A according to some embodiments.

FIG. 2B is a flowchart of a method for generating a gaze point modulation plane by the drive controller circuit 120 of FIG. 1A according to some embodiments.

2C is a schematic flowchart of a method or procedure for writing the gaze point modulated plane data into the display pixel array by the gaze point modulated display device 130 of FIG. 1A according to some embodiments.

2D is a schematic flowchart of a method or procedure for writing gaze point image frame data into the display pixel array of the gaze point grayscale display device 162 of FIG. 1B according to some embodiments.

FIG. 3A is a schematic flowchart of a method or procedure for writing a modulation plane of gaze point data into the pixel array of the gaze point modulation display device 130 of FIG. 1A in the center detail mode operation according to some embodiments. There are four blocks in it.

FIG. 3B is a schematic flowchart of a method or procedure for writing a modulation plane of gaze point data into the pixel array of the gaze point modulation display device 130 of FIG. 1A in the peripheral detail mode operation according to some embodiments. There are four blocks in it.

FIG. 4A is a multi-level block diagram of the expansion of the fixation point data in the method of FIG. 3A according to some embodiments, showing the contents of the fixation point data block and the line buffer.

FIG. 4B is a continuous multi-level block diagram of the expansion of the gaze point data of FIG. 4A according to some embodiments.

FIG. 4C is a schematic diagram showing the continuous multi-steps of the expansion of the macro pixel data of FIG. 4A.

FIG. 5 is a timing diagram of a block sequential frame or plane according to an embodiment of the present invention, showing the block buffer writing and reading sequence.

FIG. 6 shows an exemplary computing device that can implement the embodiments described herein.

FIG. 7 is a timing diagram of transport illumination according to some embodiments, showing color sequence images and planes related to buffer types.

FIG. 8A shows the data format of the gaze point image in the host memory according to some embodiments.

FIG. 8B shows a data format for image frames or modulation planes according to some embodiments, in which the data is sent sequentially through blocks with zone pads.

FIG. 8C shows a data format for an image frame or a modulation plane according to some embodiments, in which the data is sent sequentially through blocks with row pads.

FIG. 8D shows a data format for image frames or modulation planes according to some embodiments, in which the data is transmitted by filling in the sequence of each block and line group row through the raster sequence and block arrangement sequence.

FIG. 8E shows a data format for image frames or modulation planes according to some embodiments, in which data is transmitted through raster sequence and JIT sequence with each block and line group row timing filling.

FIG. 9A shows the physical layout of the column multi-drive configuration according to some embodiments.

FIG. 9B illustrates the number of row writes required for the multi-drive arrangement of FIG. 9A according to some embodiments, which are used for setting conditions of a specific line group and grouping at the same time.

Figure 10 shows the diameter, field of view, and cone density of each area of the human eyeball about the fovea that can be obtained from public sources.

100: Gaze point electromagnetic radiation modulation system

103: Gaze point image frame

105: input data source

107: Tracking Logic

110: processor circuit

111: block definition module

112: Gaze point rendering module

114: Gaze point image

115: image protocol encoding

120: drive controller circuit

122: memory

123: Image protocol decoding

124: Conversion unit (image transposition plane)

125: plane agreement coding

126: Conversion unit (bit plane to modulation plane)

127: Modulation Plane

130: Modulation device circuit

132: Memory

133: flat protocol decoding

140: display circuit

142: Control Unit

144: pixel array circuit

146: Display

150: Raster logic

152: Wire group collection circuit

154: write logic directly

156: line buffer

158: Row to Column

Z0~Zn: block

Claims (20)

A gaze point light modulation system, comprising: a processor coupled to receive an input image data and a gaze point area definition data to generate a gaze point image frame, the gaze point image frame having a header Packet data, the header packet data indicates a first block with a first resolution and a second block with a second resolution, wherein the second resolution is smaller than the first resolution, and wherein At least one of the first block and the second block is compressed based on a macropixel ratio; a drive controller circuit coupled to receive the gaze point image frame from the processor, and A plurality of modulation planes are generated based on at least partly looking at the bit plane data; and a modulation device having a plurality of pixels including at least one giant pixel, the modulation device is coupled to the drive controller circuit, and the drive control The device circuit receives the modulation planes, and each of the modulation planes is expanded based on the header packet data. The gaze point light modulation system according to claim 1, wherein for the second block with the second resolution, a single bit in a modulation plane represents a giant pixel of the modulation device, and The single bit is copied to a subset of display pixels defined by the macro pixel ratio. The gaze point light modulation system according to claim 1, wherein the modulation device is a display. The gaze point light modulation system according to claim 1, wherein the modulation device is a Liquid Crystal on Silicon (LCOS) display. The gaze point light modulation system according to claim 1, wherein the modulation device includes a decoding logic module coupled to receive the modulation plane. The gaze point light modulation system according to claim 5, wherein the modulation device further includes a raster logic, and wherein the decoding logic module parses the header packet data and the raster logic. The gaze point light modulation system according to claim 6, wherein the decoding logic module generates a data set based on the ratio of the header packet data and the macro pixels in the modulation planes. The gaze point light modulation system according to claim 1, further comprising: a tracking logic module coupled to the processor, wherein the tracking logic module senses retinal gaze data and head position data of a user, And generating gaze point block data corresponds to the sensed retinal gaze data and the head position data. The gaze point light modulation system according to claim 8, wherein the processor further includes: a gaze point rendering module coupled to the tracking logic module and receiving the sensed retinal gaze data and the head position data , And use a fixation point rendering algorithm to determine the size and position of each block. The gaze point light modulation system according to claim 9, wherein the gaze point rendering module is based on a total field of view data, an optical system distortion data, a gaze point visual acuity data, a tracking logic data allowable value, a Track at least one of logical data delay and a moving data rate, and use the viewpoint rendering algorithm to determine the size and position of each block. The gaze point light modulation system according to claim 1, wherein the header packet data includes: a resolution sequence trigger bit, enabling a center detail mode and a peripheral detail mode, wherein when the center detail mode is activated When the gaze point image frame includes a plurality of concentric blocks including the first and second blocks, and the block with the highest resolution of the first and second blocks is located at the center of a user’s gaze point, Therefore, the resolution of a block adjacent to at least one of the first and second blocks is lower than the full resolution of the other adjacent blocks by a predetermined value, and the resolution of one of the concentric blocks Starting from the user’s gaze point, it decreases in descending order; and when the peripheral detail mode is activated, the gaze point image frame includes a plurality of concentric blocks with the highest resolution located in the center of the concentric blocks. Surrounding areas, so that the resolution of the second block is lower than the first block by a predetermined value, and the resolution of each concentric inner block decreases in descending order; a transmission mode switching bit enables a raster sequence mode and a Block sequential mode, where when the raster sequential mode is active, data transmission includes multiple line groups, these line groups represent data lines from the concentric blocks, corresponding to the display sequence of an original image, and When the block sequence mode is active, the data transmission includes the complete transmission of each of the concentric blocks before the data transmission of an adjacent block; a block number segment defines the number of concentric blocks Quantity; a block size segment, which defines the horizontal and vertical size of each of the concentric blocks; a block offset segment, which defines the horizontal and vertical offsets of each of the concentric blocks; and multiple Display parameters. The gaze point light modulation system according to claim 11, wherein the display parameters include: a character length segment, which defines a plurality of pixel bits transmitted in a timing cycle related to the drive controller circuit; x offset size segment, in each horizontal offset least significant bit (Least Significant Bit, LSB) to define multiple pixel bits; a line group size segment, define the maximum number of lines to be written at the same time; one line time period, define the write Multiple timing segments required to enter a row; and a dual-column drive mode indicator that can write two rows at the same time. A method for generating a gaze point image on a display screen includes: receiving an input image data and a plurality of protocol parameters, wherein the input image data is associated with the gaze point image and a plurality of gaze point area parameters; based on The input image data and the gaze point area parameters generate rendered gaze point image data; based on the rendered gaze point image data and the protocol parameters, a gaze point image frame is generated, wherein the gaze point image The frame includes header packet data; and the gaze point image frame is transmitted to at least one of a grayscale device and a modulation device. The method according to claim 13, wherein the gaze point image frame indicates two or more concentric blocks with different resolutions, so that each concentric block is composed of a plurality of macro pixels and a corresponding macro pixel ratio Defined. The method according to claim 13, wherein at least one of the grayscale device and the modulation device includes a decoding logic and a raster logic. The method according to claim 15, wherein at least one of the grayscale device and the modulation device includes a plurality of pixels, and the method further includes: based on the header packet data and the corresponding ratio of each of the macro pixels Using the raster logic to generate the gaze point image on at least some pixels of at least one of the grayscale device and the modulation device. The method according to claim 16, wherein for the gaze area with reduced resolution, a single bit of a plurality of related macro pixels is copied based on the corresponding macro pixel ratio associated with each block To a subset of multiple display pixel arrays. The method according to claim 13, wherein receiving the input image data includes receiving tracking data in real time based on at least one of the retina, the user's gaze direction data and position data, and at least one gaze direction. The method according to claim 18, wherein the gaze point image data is based on the input image data, the gaze point area parameters, and the tracking data. The method according to claim 13, wherein generating a rendered gaze point image includes: using gaze point rendering 3D to 2D rendering technology or other gaze point rendering technology known in the industry to generate a plurality of image giant pixels, The input image data, the observer's gaze direction data, the vantage point data, and a plurality of fixation point area parameters are used to represent the projected text or graphics in the fixation point image space.

Images