WO2024066659A1 - Image processor, processing method, storage medium and augmented reality display apparatus - Google Patents

Abstract

Provided in the present invention are an image processor, an image processing method, a storage medium and an augmented reality display apparatus. The image processor comprises a display pipeline. A software processing unit and at least one image processing hardening unit are integrated in the display pipeline, which is configured to: acquire an eye movement signal of a user and a first image to be processed; by using a first software configured in the software processing unit, and the at least one image processing hardening unit, perform distortion correction processing on the first image; according to the eye movement signal, perform compression processing on the first image by using a second software configured in the software processing unit, and the at least one image processing hardening unit; and transmit to a display terminal a second image obtained by means of the distortion correction processing and the compression processing, so as to perform image display.

Description

Image processor, processing method, storage medium and extended reality display device

This application claims priority to a patent application with a filing date of September 27, 2022, Chinese application number 202211184446.5, and titled “Image Processor, Processing Method, Storage Medium and Extended Reality Display Device”.

Technical Field

The present invention relates to an extended reality display technology, and in particular to an image processor, an image processing method, a computer-readable storage medium, and an extended reality display device.

Background technique

Extended Reality (XR) display technology refers to the technology that combines the real and the virtual through computers to create a virtual environment that can be interacted with humans and machines, including but not limited to Augmented Reality (AR) display technology, Virtual Reality (VR) display technology, and Mixed Reality (MR) display technology. By integrating these three visual interaction technologies, Extended Reality display technology can bring the experiencer an immersive feeling of seamless transition between the virtual world and the real world.

In response to the high-resolution display requirements in the XR field, the display resolution of existing XR devices generally cannot reach the resolution required for 4K display. When the resolution of the human eye's gaze range is less than 40 pixels per degree, users will clearly see the pixels and their boundaries, which will affect the display effect.

In order to overcome the above-mentioned defects of the prior art, the art is in urgent need of an image processing technology that can meet the high-resolution display requirements of XR display devices based on limited software and hardware resources.

Summary of the invention

A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or decisive elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

In order to overcome the above-mentioned defects in the prior art, the present invention provides an image processor, an image processing method, a computer-readable storage medium, and an extended reality display device, which can provide a compression processing function of high-resolution display of a local area according to the user's eye movement signal, and improve the storage, processing and transmission efficiency of data by setting and reusing image processing hardening units, thereby based on limited software and hardware software resources to meet the high-resolution display requirements of XR display devices.

Specifically, the image processor provided according to the first aspect of the present invention includes a display pipeline. The display pipeline integrates a software processing unit and at least one image processing hardening unit, and is configured to: obtain the user's eye movement signal and the first image to be processed; use the first software configured in the software processing unit and the at least one image processing hardening unit to perform distortion correction processing on the first image; according to the eye movement signal, use the second software configured in the software processing unit and the at least one image processing hardening unit to compress the first image; and transmit the second image obtained after the distortion correction processing and the compression processing to the display terminal for image display.

Furthermore, in some embodiments of the present invention, the at least one image processing hardening unit is selected from at least one of a cache memory, a weighted summation circuit, a mean value calculation circuit, a filtering circuit, and a pixel position relationship mapping circuit.

Furthermore, in some embodiments of the present invention, the display pipeline is connected to a main processor of the augmented reality display device. The step of obtaining the user's eye movement signal and the first image to be processed includes: obtaining a real scene image and/or a virtual image compressed by gaze point rendering via the main processor, wherein the gaze point rendering compression is implemented based on the eye movement signal.

Furthermore, in some embodiments of the present invention, the image processor also includes display driver software and/or a firmware computing platform. The display driver software and/or the firmware computing platform are respectively connected to the eye tracker and the display pipeline. The step of obtaining the user's eye movement signal and the first image to be processed includes: obtaining the eye movement signal from the eye tracker via the display driver software and/or the firmware computing platform, and performing eye tracking calculations to determine the gaze point position; updating the gaze point information according to the gaze point position to construct an updated compression model; and transmitting the compression parameters of the updated compression model to the display pipeline.

Furthermore, in some embodiments of the present invention, the step of compressing the first image according to the eye movement signal using the second software configured in the software processing unit and the at least one image processing hardening unit includes: determining the coordinates of multiple partitions about the gaze point position and the downsampling ratio of each of the partitions according to the compression parameters of the updated compression model; and performing the compression processing on the first image according to the coordinates and the downsampling ratio of each of the partitions using the second software configured in the software processing unit in cooperation with the at least one image processing hardening unit.

Furthermore, in some embodiments of the present invention, the software processing unit is further configured with a third software The display pipeline is further configured to: determine the coordinates of a plurality of partitions about the gaze point position and the upsampling ratio of each of the partitions according to the compression parameters of the updated compression model; and perform super-resolution processing on the first image by using the third software configured in the software processing unit in cooperation with the at least one image processing hardening unit according to the coordinates and the upsampling ratio of each of the partitions.

Further, in some embodiments of the present invention, the plurality of partitions have the same and/or different downsampling ratios, and/or the plurality of partitions have the same and/or different upsampling ratios.

Further, in some embodiments of the present invention, the display terminal is configured with a decompression module. The display pipeline and the display driver software and/or the firmware computing power platform are respectively connected to the decompression module. The step of transmitting the second image obtained after the distortion correction processing and the compression processing to the display terminal for image display includes: transmitting the second image after the distortion correction processing and the compression processing to the decompression module via the display pipeline; transmitting the eye movement signal, the gaze point position and/or the compression parameters of the updated compression model to the decompression module via the display driver software and/or the firmware computing power platform; using the decompression module, according to the eye movement signal, the gaze point position and/or the compression parameters of the updated compression model, the second image after the distortion correction processing and the compression processing is decompressed to obtain a third image; and according to the third image after the decompression processing, the image is displayed on the display terminal.

Furthermore, in some embodiments of the present invention, a decompression module is also integrated in the display pipeline. The display driver software and/or the firmware computing power platform are connected to the decompression module. The software processing unit and the at least one image processing hardening unit are connected to the display terminal via the decompression module. The step of transmitting the second image obtained after the distortion correction processing and the compression processing to the display terminal for image display includes: transmitting the second image after the distortion correction processing and the compression processing to the decompression module via the software processing unit and the at least one image processing hardening unit; transmitting the compression parameters of the updated compression model to the decompression module via the display driver software and/or the firmware computing power platform; using the decompression module, according to the compression parameters of the updated compression model, decompressing the second image after the distortion correction processing and the compression processing to obtain a third image; and transmitting the third image after the decompression processing to the display terminal for image display.

In addition, the image processing method provided according to the second aspect of the present invention comprises the following steps: acquiring an eye movement signal and a first image to be processed; using a second image processing unit configured in a software processing unit integrated in a display pipeline; a software, and at least one integrated image processing hardening unit, to perform distortion correction processing on the first image; based on the eye movement signal, using the second software configured in the software processing unit, and the at least one image processing hardening unit, to compress the first image; and transmitting the second image that has undergone the distortion correction processing and the compression processing to a display terminal for image display.

In addition, the computer-readable storage medium provided in the third aspect of the present invention stores computer instructions, which, when executed by a processor, implement the image processing method provided in the second aspect of the present invention.

In addition, the above-mentioned extended reality display device provided according to the fourth aspect of the present invention includes an eye tracker, a main processor, a coprocessor and a display terminal. The eye tracker is used to collect the user's eye movement signals. The main processor outputs a first image that has been or has not been compressed by gaze point rendering, wherein the gaze point rendering compression is implemented based on the eye movement signal. The coprocessor can use the above-mentioned image processor provided by the first aspect of the present invention, wherein the image processor is respectively connected to the eye tracker and the main processor to obtain the first image and the eye movement signal. The display terminal is connected to the image processor to obtain and display the second image that has been processed by the image processor for distortion correction and compression.

BRIEF DESCRIPTION OF THE DRAWINGS

The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

FIG. 1 shows a schematic diagram of an extended reality display device according to some embodiments of the present invention.

FIG. 2 is a schematic flow chart showing an image processing method according to some embodiments of the present invention.

FIG3 is a schematic diagram showing a process of recompressing an image according to some embodiments of the present invention.

FIG. 4 shows a schematic diagram of local image compression processing provided according to some embodiments of the present invention.

Detailed ways

The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this embodiment. On the contrary, the purpose of introducing the invention in conjunction with the embodiment is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many The present invention may also be implemented without using these details. In addition, in order to avoid confusion or blurring the key points of the present invention, some specific details will be omitted in the description.

In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and/or parts, these components, regions, layers and/or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and/or parts. Therefore, the first component, region, layer and/or part discussed below may be referred to as a second component, region, layer and/or part without departing from some embodiments of the present invention.

As mentioned above, in response to the high-resolution display requirements in the field of Extended Reality (XR), the display resolution of existing XR devices generally cannot reach the resolution required for 4K display. When the resolution of the human eye's gaze range is less than 40 pixels per degree, the user will clearly see the pixels and their boundaries, which will affect the display effect. In response to the demand for improving display resolution, some improved technologies for multi-screen superposition or multi-image nested output have been proposed in the field. However, based on these improved technologies, the solution of using mechanical structures to manipulate multiple screens to achieve high-definition gaze point rendering has the defects of high hardware complexity and occupying a large volume. The solution of using multi-image nested transmission for gaze point compression will produce obvious jagged and flickering phenomena at the boundary between high-definition images and low-definition images, which will also affect the display effect.

In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processor, an image processing method, a computer-readable storage medium, and an extended reality display device, which can provide compression processing functions for high-resolution display of local areas according to the user's eye movement signals, and improve the storage, processing and transmission efficiency of data by setting and reusing image processing hardening units, thereby meeting the high-resolution display requirements of XR display devices based on limited software and hardware resources.

In some non-limiting embodiments, the image processing method provided in the second aspect of the present invention can be implemented via the image processor provided in the first aspect of the present invention. Specifically, the image processor can be independently configured in the extended reality display device provided in the fourth aspect of the present invention in the form of a coprocessor chip, or can be integrated into the main processors such as the central processing unit (CPU) and the graphics processing unit (GPU) of the extended reality display device provided in the fourth aspect of the present invention in the form of software programs and hardware units.

Furthermore, the image processor provided in the first aspect of the present invention may be configured with or connected to a processing unit and a storage unit of a software program. The storage unit includes but is not limited to the computer-readable storage medium provided in the third aspect of the present invention, on which computer instructions are stored. The processing unit is connected to the storage unit and is configured to execute the computer instructions stored in the storage unit to implement the image processing method provided in the second aspect of the present invention.

The following will describe the working principles of the above-mentioned image processor and extended reality display device in conjunction with some embodiments of the image processing method. In some non-limiting embodiments, the extended reality display device can adopt a system architecture of a coprocessor. Those skilled in the art will understand that these embodiments of the image processing method are only some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the image processor and the extended reality display device. Similarly, the image processor and the extended reality display device are only a non-limiting implementation method provided by the present invention, and do not constitute a limitation on the execution subject of each step in these image processing methods.

Please refer to Figure 1 and Figure 2. Figure 1 shows a schematic diagram of an extended reality display device provided according to some embodiments of the present invention. Figure 2 shows a flow chart of an image processing method provided according to some embodiments of the present invention.

As shown in FIG1 , in some embodiments of the present invention, the extended reality display device may be configured with a coprocessor 10, an eye tracker 20, a main processor 30, a display driver chip 40, and a display terminal 50. Furthermore, the coprocessor 10 may select the above-mentioned image processor provided in the first aspect of the present invention, which is configured with a display pipeline. The display pipeline integrates a software processing unit and at least one image processing hardening unit, and is configured to use different software programs and the same image processing hardening unit to perform image processing processes such as image distortion correction and image compression, thereby improving the storage, processing and transmission efficiency of data, and meeting the high-resolution display requirements of the XR display device based on limited software and hardware resources. The eye tracker 20 is used to collect eye movement signals such as the gaze position and gaze direction of the user. The main processor 30 is selected from common image processors such as a central processing unit (CPU) and a graphics processing unit (GPU), and is used to output a first image to be processed by the coprocessor 10. The display driver chip 40 can be integrated in the display terminal 50, and is used to obtain the second image processed by the coprocessor 10, and display the image via the pixel array circuit and OLED/LED pixel array configured in the display terminal 50.

Specifically, the software processing unit configured in the display pipeline of the coprocessor 10 includes but is not limited to a distortion correction unit and/or an image compression unit. The image processing hardening unit configured in the display pipeline can be selected from at least one of a transistor-level cache memory, a weighted summation circuit, a mean value calculation circuit, a filtering circuit, and a pixel position relationship mapping circuit, and is used to cache multiple pixel data in the first image of the current frame and/or several previous historical frames, and/or perform weighted summation, mean value calculation, filtering, and/or pixel position relationship mapping hardening calculations on these pixel data.

As shown in FIG2 , during the image processing, the display pipeline may first obtain the user's eye movement signal and the first image to be processed. Here, the eye movement signal includes but is not limited to data such as the user's eye deflection angle, gaze position, and gaze direction. The display pipeline may be directly connected to the eye tracker 20 and directly obtain the eye movement signal from the eye tracker 20, or it may be indirectly connected to the eye tracker 20 via an image processing unit such as the GPU of the main processor 30, and the user's eye movement signal may be indirectly obtained synchronously via the main processor 30. In addition, for an embodiment of a mixed reality (MR) display, the first image may be an original virtual rendering image generated by an image processing unit such as the GPU of the main processor 30.

Furthermore, in some preferred embodiments, the main processor 30 may also be connected to the eye tracker 20, and be configured to obtain the user's eye movement signal from the eye tracker 20, and first perform gaze point rendering compression on the generated original virtual rendering image according to the eye movement signal, and then send the compressed image after gaze point rendering compression to the coprocessor 10, so as to reduce the data transmission load and data processing load of the entire architecture.

As shown in Figures 1 and 2, after acquiring the first image and the user's eye movement signal, the display pipeline can use the image distortion correction unit (i.e., the first software) configured therein and at least one image processing hardening unit integrated therein to perform distortion correction processing on the acquired first image.

Specifically, in response to acquiring the first image to be processed, the image distortion correction unit (i.e., the first software) may first determine the pixel data and processing parameters required for the image distortion correction process. In some embodiments, the pixel data may preferably be determined based on the user's eye movement signal. Thereafter, the image distortion correction unit (i.e., the first software) may obtain the processing parameters from each corresponding memory and store the processing parameters from each corresponding cache. The memory is used to obtain pixel cache data required for image distortion correction processing, and the processing parameters and pixel cache data are sequentially input into one or more of the weighted summation circuit, the mean value calculation circuit, the filtering circuit, and the pixel position relationship mapping circuit, and the pixel cache data are subjected to weighted summation, mean value calculation, filtering, and/or pixel position relationship mapping hardening calculations to obtain corresponding hardening calculation results. After that, the image distortion correction unit (i.e., the first software) can obtain the image after distortion correction processing through software operations such as data arrangement and value assignment.

Furthermore, in some embodiments as shown in FIG. 1 , the coprocessor 10 may also be preferably configured with a display driver software and/or firmware computing power platform. The display driver software and/or firmware computing power platform are respectively connected to the eye tracker 20 and the display pipeline. In the process of compressing the first image, the coprocessor 10 may first obtain the user's eye movement signal from the eye tracker 20 via the display driver software and/or firmware computing power platform, and perform eye tracking calculations to determine the gaze point position. Afterwards, the display driver software and/or firmware computing power platform may update the gaze point information within the system according to the gaze point position to construct an updated compression model, and transmit the compression parameters of the updated compression model to the display pipeline for it to compress the first image.

As shown in Figure 2, in response to acquiring the first image and the eye movement signal (or the compression parameters associated with the user's eye movement signal), the display pipeline can compress the first image according to the eye movement signal using the image compression unit configured therein (i.e., the second software) and at least one image processing hardening unit integrated therein.

Specifically, in response to obtaining the compression parameters associated with the user's eye movement signal and the first image to be processed, the image compression unit (i.e., the second software) can first divide the first image into multiple partitions based on the user's gaze point position according to the compression parameters of the updated compression model, and respectively determine the coordinate range of each partition and the downsampling magnification of each non-focused partition away from the gaze point position. Afterwards, the image compression unit (i.e., the second software) can obtain the processing parameters of the downsampling operation from each corresponding memory according to the coordinates of each pixel in each partition, and obtain the pixel cache data required for the downsampling process from each corresponding cache memory. Afterwards, the image compression unit (i.e., the second software) can sequentially input the obtained processing parameters and the pixel cache data of each pixel into one or more of the weighted summation circuit, the mean calculation circuit, the filtering circuit, and the pixel position relationship mapping circuit, and perform hardening calculations of weighted summation, mean calculation, filtering, and/or pixel position relationship mapping on these pixel cache data to obtain the first hardening calculation result based on the downsampling process. Afterwards, the image compression unit (ie, the second software) can obtain the down-sampled compressed image through software operations such as data arrangement and value assignment.

In addition, in some embodiments of the present invention, the display pipeline of the coprocessor 10 may preferably be configured with a super-resolution unit (ie, the third software), so as to perform re-compression processing on the first image together with the image compression unit (ie, the second software).

Specifically, in the process of recompressing the first image, in response to obtaining the compression parameters associated with the user's eye movement signal and the first image to be processed, the super-resolution unit (i.e., the third software) can first divide the first image into multiple partitions based on the user's gaze point position according to the compression parameters of the updated compression model, and respectively determine the coordinate range of each partition, as well as the upsampling magnification of each focus partition containing the adjacent gaze point position. Afterwards, the super-resolution unit (i.e., the third software) can obtain the processing parameters of the downsampling operation from each corresponding memory according to the coordinates of each pixel in each partition, and obtain the pixel cache data required for super-resolution processing from each corresponding cache memory. Here, the pixel cache data includes but is not limited to the cache data of at least one nearby pixel of the current frame, and the cache data of these nearby pixels in at least one historical frame. Afterwards, the super-resolution unit (i.e., the third software) can sequentially input the acquired processing parameters and the pixel cache data of each pixel into one or more of the weighted summation circuit, the mean value calculation circuit, the filtering circuit, and the pixel position relationship mapping circuit, and perform the weighted summation, mean value calculation, filtering, and/or pixel position relationship mapping hardening calculations on the pixel cache data to obtain a second hardening calculation result based on the super-resolution processing. Afterwards, the super-resolution unit (i.e., the third software) can summarize the second hardening calculation result and the first hardening calculation result, and obtain the heavily compressed image shown in FIG4 through software operations such as data arrangement and value assignment.

As shown in FIG4 , after the recompression processing based on the eye movement information, at least one focus partition in the first image containing a position close to the gaze point is super-resolved based on the same and/or different upsampling ratios, thereby achieving a 4K display equivalent resolution of 40 pixels per degree in these focus partitions. At the same time, at least one non-focus partition in the first image far from the gaze point is compressed based on the same and/or different downsampling ratios, thereby reducing the data processing load and/or data transmission load of these non-focus partitions.

Compared with the pure software solution of distortion correction and image compression that needs to calculate and cache the pixel data of each relevant pixel frame by frame and partition by partition, the present invention effectively improves the reuse rate of pixel cache data of each frame and each partition by designing and reusing the cache memory, weighted summation circuit, mean value calculation circuit, filtering circuit, and pixel position relationship mapping circuit, and effectively reduces the data processing load of the software unit, thereby reducing the data processing and transmission load of the coprocessor 10 as a whole, so as to facilitate the coprocessor 10 to realize the image processing based on limited software. Hardware resources to meet the high-resolution display requirements of XR display devices.

As shown in FIG. 2 , after obtaining the second image that has been subjected to distortion correction and compression processing by the coprocessor 10 , the coprocessor 10 may transmit the second image to the display terminal 50 for high-resolution display of the augmented reality image.

Specifically, for the embodiment in which the main processor 30 sends the first image compressed by gaze point rendering to the coprocessor 10, a decompression module may be preferably integrated in the display pipeline of the coprocessor 10. The above-mentioned display driver software and/or firmware computing power platform is connected to the decompression module. The software processing unit and at least one image processing hardening unit configured in the display pipeline are also connected to the display terminal 50 via the decompression module. In this way, in the process of transmitting the second image to the display terminal 50, the decompression module configured in the coprocessor 10 can first obtain the second image after distortion correction processing and compression processing through the software processing unit and at least one image processing hardening unit integrated in the display pipeline, and then obtain the compression parameters of the updated compression model through the above-mentioned display driver software and/or firmware computing power platform, so as to decompress the second image after distortion correction processing and compression processing according to the compression parameters of the updated compression model to obtain the third image. Afterwards, the coprocessor 10 can transmit the decompressed third image to the display terminal 50 for high-resolution display of the extended reality image.

Further, for the embodiment in which the main processor 30 sends the first image compressed by gaze point rendering to the coprocessor 10, the above-mentioned decompression module can also be preferably configured on the display terminal 50 or its display driver chip 40. Taking the decompression module configured on the display driver chip 40 as an example, the decompression module can be connected to the display pipeline, display driver software and/or firmware computing platform of the coprocessor 10 respectively. In the process of transmitting the second image to the display terminal 50, the decompression module configured on the display driver chip 40 can first obtain the second image after distortion correction processing and compression processing through the software processing unit and at least one image processing hardening unit integrated in the display pipeline, and then obtain the user's eye movement signal, gaze point position and/or updated compression model compression parameters through the above-mentioned display driver software and/or firmware computing platform, so as to decompress the second image after distortion correction processing and compression processing according to the eye movement signal, gaze point position and/or updated compression model compression parameters to obtain the third image. Afterwards, the display driver chip 40 can directly drive the pixel array circuit of the display terminal 50 such as the OLED display screen/LED display screen according to the decompressed third image, so as to display the extended reality image with high resolution on the display terminal 50 .

Furthermore, in some embodiments, the decompression module can be configured in a pixel array circuit of the display terminal 50. In this way, the image data transmitted in the entire architecture of the augmented reality display device is The data is compressed by gaze point rendering, which can further reduce the data transmission load and data processing load of the entire architecture. The working principle of the decompression module of the pixel array circuit configured in the display terminal 50 is similar to that of the embodiment configured in the display driver chip 40, which will not be repeated here.

Finally, those skilled in the art will appreciate that the system architecture of the extended reality display device using the coprocessor 10 is only a non-limiting implementation provided by the present invention, which is intended to clearly demonstrate the main concept of the present invention and provide a specific solution that is easy for the public to implement, rather than to limit the scope of protection of the present invention.

Optionally, in other embodiments, the above-mentioned image processor provided in the first aspect of the present invention can also be integrated into the main processor units such as the central processing unit (CPU) and graphics processing unit (GPU) of the above-mentioned extended reality display device provided in the fourth aspect of the present invention in the form of software programs and hardware units to achieve the same technical effects, which will not be repeated here.

Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and/or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician can implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented using a general purpose processor, an NPU AI network model computing acceleration processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates, or chips. The processor may be implemented or executed by a combination of a processor, a processor logic, a discrete hardware component, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative solution, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from/to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims (12)

1. An image processor, characterized in that it comprises a display pipeline, wherein the display pipeline integrates a software processing unit and at least one image processing hardening unit, and is configured as follows:

   Acquire the user's eye movement signal and the first image to be processed;

   Using the first software configured in the software processing unit and the at least one image processing hardening unit to perform distortion correction processing on the first image;

   According to the eye movement signal, using the second software configured in the software processing unit and the at least one image processing hardening unit to compress the first image; and

   The second image obtained through the distortion correction process and the compression process is transmitted to a display terminal for image display.
2. The image processor according to claim 1, characterized in that the at least one image processing hardening unit is selected from at least one of a cache memory, a weighted summation circuit, a mean value calculation circuit, a filtering circuit, and a pixel position relationship mapping circuit.
3. The image processor according to claim 1, wherein the display pipeline is connected to a main processor of the extended reality display device, and the step of obtaining the user's eye movement signal and the first image to be processed comprises:

   A real scene image and/or a virtual image compressed by gaze point rendering is obtained via the main processor, wherein the gaze point rendering compression is implemented based on the eye movement signal.
4. The image processor according to claim 1, further comprising display driver software and/or a firmware computing platform, wherein the display driver software and/or the firmware computing platform are respectively connected to an eye tracker and the display pipeline, and the step of obtaining the user's eye movement signal and the first image to be processed comprises:

   Acquiring the eye movement signal from the eye tracker via the display driver software and/or the firmware computing platform, and performing eye movement tracking calculation to determine the gaze point position;

   updating gaze point information according to the gaze point position to construct an updated compression model; and

   The compression parameters of the updated compression model are transmitted to the display pipeline.
5. The image processor according to claim 4, characterized in that the step of compressing the first image according to the eye movement signal using the second software configured in the software processing unit and the at least one image processing hardening unit comprises:

   Determining coordinates of a plurality of partitions about the gaze point position and a downsampling ratio of each of the partitions according to the compression parameters of the updated compression model; and

   According to the coordinates and downsampling ratio of each partition, the software processing unit is configured to The second software cooperates with the at least one image processing hardening unit to perform the compression process on the first image.
6. The image processor according to claim 5, characterized in that the software processing unit is further configured with third software, and the display pipeline is further configured as:

   Determining coordinates of a plurality of partitions about the gaze point position and an upsampling ratio of each of the partitions according to the compression parameters of the updated compression model; and

   According to the coordinates of each of the partitions and the upsampling ratio, the first image is subjected to super-resolution processing by using the third software configured in the software processing unit in cooperation with the at least one image processing hardening unit.
7. The image processor according to claim 5 or 6, characterized in that the multiple partitions have the same and/or different downsampling ratios, and/or the multiple partitions have the same and/or different upsampling ratios.
8. The image processor according to claim 4 or 5, characterized in that the display terminal is configured with a decompression module, the display pipeline and the display driver software and/or the firmware computing power platform are respectively connected to the decompression module, and the step of transmitting the second image obtained after the distortion correction processing and the compression processing to the display terminal for image display comprises:

   Transmitting the second image after the distortion correction processing and the compression processing to the decompression module via the display pipeline;

   Transmitting the eye movement signal, the gaze point position and/or the compression parameters of the updated compression model to the decompression module via the display driver software and/or the firmware computing platform;

   Using the decompression module, decompressing the second image after the distortion correction and compression processing according to the eye movement signal, the gaze point position and/or the compression parameter of the updated compression model to obtain a third image; and

   An image is displayed on the display terminal according to the decompressed third image.
9. The image processor according to claim 4 or 5, characterized in that a decompression module is also integrated in the display pipeline, the display driver software and/or the firmware computing power platform is connected to the decompression module, the software processing unit and the at least one image processing hardening unit are connected to the display terminal via the decompression module, and the step of transmitting the second image obtained after the distortion correction processing and the compression processing to the display terminal for image display comprises:

   Transmitting the second image after the distortion correction processing and the compression processing to the decompression module via the software processing unit and the at least one image processing hardening unit;

   Transmitting compression parameters of the updated compression model to the decompression module via the display driver software and/or the firmware computing platform;

   Using the decompression module, according to the compression parameters of the updated compression model, the decompressing the second image processed by the correction and compression to obtain a third image; and

   The decompressed third image is transmitted to the display terminal for image display.
10. An image processing method, characterized in that it comprises the following steps:

    Acquiring an eye movement signal and a first image to be processed;

    Using first software configured in a software processing unit integrated in a display pipeline and at least one integrated image processing hardening unit to perform distortion correction processing on the first image;

    According to the eye movement signal, using the second software configured in the software processing unit and the at least one image processing hardening unit to compress the first image; and

    The second image that has undergone the distortion correction processing and the compression processing is transmitted to a display terminal for image display.
11. A computer-readable storage medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the image processing method according to claim 10 is implemented.
12. An extended reality display device, comprising:

    Eye tracker, used to collect user's eye movement signals;

    A main processor outputs a first image that has been subjected to or has not been subjected to gaze point rendering compression, wherein the gaze point rendering compression is implemented based on the eye movement signal;

    The image processor according to any one of claims 1 to 9, wherein the image processor is connected to the eye tracker and the main processor respectively to obtain the first image and the eye movement signal; and

    The display terminal is connected to the image processor to obtain and display the second image that has been subjected to the distortion correction and compression processing by the image processor.