TWI879397B - Image processing system and method based on center concavity principle, and storage medium - Google Patents

Abstract

The present invention provides an image processing system and method, a storage medium, and an extended reality display apparatus. The image processing system includes a display pipeline and an image signal processing pipeline. The display pipeline acquires an eye-movement signal of a user, and performs, according to the eye-movement signal, a first center concavity processing. The display pipeline also generates, according to the eye-movement signal, a frame synchronization signal, and transmits the frame synchronization signal to the image signal processing pipeline for the image signal processing pipeline to perform a second center concavity processing according to the frame synchronization signal. By adopting this configuration, the image processing system can comprehensively improve an image quality, power consumption, real-time performance, and user experience of an image processing through a joint interaction of the display pipeline and the image signal processing pipeline and a reusing of the display pipeline and a processing flow, thereby comprehensively improving a current situation of high frame rate display computing power shortage in augmented reality XR.

Description

Image processing system, method and storage medium based on fovea principle

The present invention relates to an extended reality display technology, and in particular to an image processing system based on the fovea principle, an image processing method based on the fovea principle, an extended reality display device, and a computer-readable storage medium.

Extended Reality (XR) display technology refers to the technology that combines the real and virtual through computers to create a virtual environment for human-computer interaction, including but not limited to augmented reality (AR) display technology, virtual reality (VR) display technology, and mixed reality (MR) display technology. By integrating these three visual interaction technologies, extended reality display technology can bring users an immersive sense of seamless transition between the virtual world and the real world.

In response to the high frame rate display requirements in the XR field, existing technologies generally use the display pipeline to process virtual rendered images or composite images after image blending based on the fovea principle, but lack joint interaction and reuse with the image signal processing (ISP) pipeline. Although it can improve the image processing quality and user experience, it has the defects of high power consumption and large delay. If the fovea image processing operation is simply transferred to the display panel drive circuit (DDIC) of the ISP pipeline for processing, there is a problem that both the ISP pipeline and the DDIC lack the display pipeline processing capability based on the gaze point, cannot support complex and detailed compensation, and the effect is limited.

In order to overcome the above-mentioned defects of existing technologies, this field urgently needs an extended reality display technology to comprehensively improve the image processing effect in terms of image quality, power consumption, real-time, user experience, etc., so as to comprehensively improve the current shortage of XR high frame rate display computing power.

The following is a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is given later.

In order to overcome the above-mentioned defects of the existing technology, the present invention provides an image processing system based on the fovea principle, an image processing method based on the fovea principle, an extended reality display device, and a computer-readable storage medium, which can comprehensively improve the image quality, power consumption, real-time performance and user experience of image processing through the data joint interaction and reuse of the display pipeline and the image signal processing pipeline, thereby comprehensively improving the current shortage of XR high frame rate display computing power.

Specifically, the above-mentioned image processing system based on the foveal principle provided according to the first aspect of the present invention includes a display pipeline and an image signal processing pipeline. The display pipeline obtains the user's eye movement signal and performs a first foveal processing according to the eye movement signal. The display pipeline also generates a frame synchronization signal according to the eye movement signal and transmits the frame synchronization signal to the image signal processing pipeline, so that the image signal processing pipeline performs a second foveal processing according to the frame synchronization signal.

Furthermore, in some embodiments of the present invention, the frame synchronization signal includes the eye movement signal, gaze point area data and/or grid matrix weight map data. The gaze point area data indicates the coordinate position of the user's gaze point on the image and/or its surrounding pixels. The grid matrix weight map data indicates the user's attention to multiple areas of the image.

Furthermore, in some embodiments of the present invention, the display pipeline is configured to: determine the user's gaze point area data according to the eye movement signal; generate the grid matrix weight map data according to the gaze point area data; perform the first foveal processing according to the grid matrix weight map data; and generate the frame synchronization signal according to the eye movement signal, the gaze point area data and/or the grid matrix weight map data.

Furthermore, in some embodiments of the present invention, the display pipeline includes a display hardening unit and a first software processing unit. The display hardening unit is connected to an eye tracker via the first software processing unit to obtain the eye movement signal, and performs the first foveal processing according to the eye movement signal. The first software processing unit is connected to the image signal processing pipeline and the display hardening unit, and synchronously aligns the frame synchronization signals of the same frame image in the image signal processing pipeline and the display hardening unit.

Furthermore, in some embodiments of the present invention, the image signal processing pipeline includes an image processing hardened unit and a second software processing unit. The image processing hardened unit is connected to the display pipeline and obtains the frame synchronization signal through the display pipeline to perform a second fovea processing based on at least one hardened computing circuit according to the frame synchronization signal. The second software processing unit is connected to the display pipeline and obtains the frame synchronization signal through the display pipeline to perform a second fovea processing based on at least one software program according to the frame synchronization signal.

Furthermore, in some embodiments of the present invention, a plurality of the hardening calculation circuits are configured in the image processing hardening unit. An independent switch is configured in at least one of the hardening calculation circuits. The hardening calculation circuit performs independent second foveal processing according to the switch signal of the corresponding switch.

Furthermore, in some embodiments of the present invention, the second software processing unit extracts the image features of the current frame image frame by frame, and configures the switch signals of each of the switches in real time according to the image features, so as to control one or more of the hardening calculation circuits in the image processing hardening unit frame by frame to perform the second foveal processing. Alternatively, the switches of each of the hardening calculation circuits in the image processing hardening unit are pre-set to corresponding switch states based on the image processing function of the image processing system, so as to fixedly control one or more of the hardening calculation circuits in the image processing hardening unit to perform the second foveal processing.

Furthermore, in some embodiments of the present invention, the display pipeline is also connected to an image rendering module, a virtual rendered image is obtained through the image rendering module, and the first foveal processing is performed on the virtual rendered image according to the eye movement signal. The image signal processing pipeline is also connected to a camera, a real scene image is obtained through the camera, and the second foveal processing is performed on the real scene image according to the frame synchronization signal. The image signal processing pipeline also transmits the real scene image that has undergone the second foveal processing to the display pipeline to perform layer blending with the virtual rendered image that has undergone or has not undergone the first foveal processing.

In addition, the above-mentioned image processing method based on the foveal principle provided according to the second aspect of the present invention includes the following steps: obtaining the user's eye movement signal; transmitting the eye movement signal to the display pipeline of the image processing system to perform the first foveal processing and generate a frame synchronization signal; and transmitting the frame synchronization signal to the image signal processing pipeline of the image processing system, so that the image signal processing pipeline performs the second foveal processing according to the frame synchronization signal.

Furthermore, in some embodiments of the present invention, the step of transmitting the eye movement signal to the display pipeline of the image processing system to perform the first foveal processing and generate the frame synchronization signal includes: determining the gaze point of the user according to the eye movement signal; generating the grid matrix weight map data according to the gaze point; performing the first foveal processing according to the grid matrix weight map data; and generating the frame synchronization signal according to the eye movement signal and/or the grid matrix weight map data.

Furthermore, in some embodiments of the present invention, the image signal processing pipeline includes an image processing hardened unit and a software processing unit. The image processing hardened unit is configured with at least one hardened computing circuit. The step of transmitting the frame synchronization signal to the image signal processing pipeline of the image processing system so that the image signal processing pipeline performs the second foveal processing according to the frame synchronization signal includes: obtaining a switch signal through the image processing hardening unit and the software processing unit; determining the hardening calculation circuit and/or software processing unit that needs to perform the second foveal processing according to the switch signal; and transmitting the frame synchronization signal to at least one hardening calculation circuit and/or software processing unit that needs to perform the second foveal processing so that the at least one hardening calculation circuit and/or software processing unit independently performs the second foveal processing.

Furthermore, in some embodiments of the present invention, a plurality of the hardening computing circuits are configured in the image processing hardening unit. The step of determining the hardening calculation circuit and/or software processing unit that needs to perform the second foveal processing according to the switch signal includes: extracting the image features of the current frame image frame by frame through the software processing unit, and configuring the switch signals of each hardening calculation circuit in real time according to the image features to control one or more hardening calculation circuits in the image processing hardening unit frame by frame to perform the second foveal processing; or presetting the switch state of each hardening calculation circuit in the image processing hardening unit based on the image processing function of the image processing system, and fixedly controlling one or more hardening calculation circuits in the image processing hardening unit to perform the second foveal processing according to the switch state of each hardening calculation circuit.

Furthermore, in some embodiments of the present invention, the display pipeline is also connected to an image rendering module, a virtual rendering image is obtained through the image rendering module, and the first foveal processing is performed on the virtual rendering image according to the eye movement signal. The image signal processing pipeline is also connected to a camera, a real scene image is obtained through the camera, and the second foveal processing is performed on the real scene image according to the frame synchronization signal. After the second foveal processing is performed, the image processing method further includes the following steps: the real scene image that has undergone the second foveal processing is transmitted from the image signal processing pipeline to the display pipeline to perform layer blending with the virtual rendering image that has undergone or has not undergone the first foveal processing.

In addition, the above-mentioned extended reality display device provided according to the third aspect of the present invention includes the above-mentioned image processing system provided by the first aspect of the present invention.

In addition, the computer-readable storage medium provided in the fourth aspect of the present invention stores computer instructions. When the computer instructions are executed by the processor, the image processing method provided in the second aspect of the present invention is implemented.

1, 2, 3: Weight

10: Image processing system

11: Display pipeline

12: Image signal processing pipeline

13: Grid matrix generation module

14: Subsequent modules

20: Eye tracker

30: Image rendering module

40: Binocular camera

111: Display hardening unit

112: First software processing unit

121: Image processing hardening unit

122: Second software processing unit

The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar drawing labels.

FIG1 shows a schematic diagram of the architecture of an image processing system provided according to some embodiments of the present invention.

Figure 2 shows a schematic diagram of the process of an image processing method provided according to some embodiments of the present invention.

FIG3 shows a schematic diagram of a grid matrix weight graph provided according to some embodiments of the present invention.

FIG4 shows a schematic diagram of the architecture of an image signal processing pipeline provided according to some embodiments of the present invention.

The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of the present invention are limited to the embodiment. On the contrary, the purpose of introducing the invention in conjunction with the embodiment is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the key points of the present invention, some specific details will be omitted in the description.

In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be a connection between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. This relative terminology is only for the convenience of explanation and does not mean that the device described must be manufactured or operated in a specific direction, so it should not be understood as a limitation of the present invention.

It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and/or parts, these components, regions, layers and/or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and/or parts. Therefore, the first component, region, layer and/or part discussed below may be referred to as the second component, region, layer and/or part without departing from some embodiments of the present invention.

As mentioned above, in response to the high frame rate display requirements in the XR field, existing technologies generally use display pipelines to perform image processing based on the fovea principle on virtual rendered images or composite images after image blending, but lack joint interaction and reuse with the image signal processing (ISP) pipeline. Although it can improve image processing quality and user experience, it has the defects of high power consumption and large delay. If the fovea image processing operation is simply transferred to the display panel drive circuit (Display Drive Integrated Circuit, DDIC) of the ISP pipeline for processing, then there is a problem that both the ISP pipeline and the DDIC lack the display pipeline processing capability based on the gaze point, cannot support complex and detailed compensation, and the effect is limited.

In order to overcome the above-mentioned defects of the existing technology, the present invention provides an image processing system based on the fovea principle, an image processing method based on the fovea principle, an extended reality display device, and a computer-readable storage medium, which can comprehensively improve the image quality, power consumption, real-time performance and user experience of image processing through the processing architecture and processing flow of data joint interaction and reuse of the display pipeline and the image signal processing pipeline, thereby comprehensively improving the current shortage of XR high frame rate display computing power.

In some non-limiting embodiments, the above-mentioned image processing method provided in the second aspect of the present invention can be implemented via the above-mentioned image processing system provided in the first aspect of the present invention. Specifically, the image processing system is configured with a memory and a processor. The memory includes but is not limited to the above-mentioned computer-readable storage medium provided in the fourth aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and is configured to execute the computer instructions stored on the memory to implement the above-mentioned image processing method provided in the second aspect of the present invention.

Furthermore, in some embodiments, the image processing system provided in the first aspect of the present invention can be configured in the extended reality display device provided in the third aspect of the present invention, and is used to reuse the image processing data of the display pipeline and/or the image signal processing pipeline under the condition of limited hardware resources, thereby saving hardware computing power and improving the hardware resource utilization of the extended reality display device while ensuring the central imaging effect.

The following will describe the working principle of the above-mentioned image processing system and extended reality display device in combination with some embodiments of the image processing method. Those skilled in the art can understand that these image processing methods are only some non-restrictive implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for public implementation, rather than to limit all functions or all working methods of the image processing system and extended reality display device. Similarly, the image processing system and extended reality display device are also only some non-restrictive implementation methods provided by the present invention, and do not constitute a limitation on the execution subject of each step in these image processing methods.

Please refer to Figure 1 and Figure 2. Figure 1 shows a schematic diagram of the architecture of an image processing system provided according to some embodiments of the present invention. Figure 2 shows a schematic diagram of the process of an image processing method provided according to some embodiments of the present invention.

As shown in FIG. 1 and FIG. 2, in some embodiments of the present invention, the image processing system 10 may be preferably configured with a display pipeline (Display Pipeline) 11 and an image signal processing pipeline (ISP Pipeline) 12. The display pipeline 11 is connected to an external device such as an eye tracker 20 to obtain the user's eye movement signal, and performs a first foveal processing on the local first image according to the obtained eye movement signal. At the same time, the display pipeline 11 is also connected to the image signal processing pipeline 12, and is also configured to generate a frame synchronization signal according to the obtained eye movement signal, and transmit the frame synchronization signal to the image signal processing pipeline 12, so that the image signal processing pipeline 12 performs a second foveal processing on the local second image according to the frame synchronization signal.

Furthermore, in the embodiment shown in FIG1 , the image processing system 10 may also be configured with at least one software module such as a grid matrix generation module 13. The grid matrix generation module 13 is selected from at least one of a digital signal processor (DSP), a microcontroller unit (MCU), and a microprocessor, and is used to generate grid matrix weight map data according to the input eye movement signal, and/or perform software calculation and/or processing on the input signal and/or data. The display pipeline 11 can be connected to the eye tracker 20 via the grid matrix generation module 13 to obtain fovea-related data such as gaze position and grid matrix weight map data, and then perform first fovea processing on the local first image based on the obtained fovea-related data.

In addition, in the embodiment shown in FIG1 , the display pipeline 11 may be preferably configured with a display hardening unit 111 and a first software processing unit 112. The display hardening unit 111 is configured with one or more transistor-level hardening calculation circuits and/or memories for hardening calculation processing and storage of input signals. The first software processing unit 112 is selected from at least one of a digital signal processor (DSP), a microcontroller (MCU), and a microprocessor, and is mainly used to calculate the gain value when adjusting the pixel brightness, calculate the pixel mapping logic relationship, calculate the configuration information of the relative relationship of the brightness of the three colors of RGB, and the format of the called image gamma data, etc., for the input signals and/or data such as the configuration registers and data processing parameters of each hardened calculation circuit and/or memory in the display hardened unit 111.

Furthermore, in some embodiments, the grid matrix generation module 13 can also be configured in the first software processing unit 112 to achieve the same effect of generating the first grid matrix weight map data according to the input eye movement signal. Here, the display hardening unit 111 can be connected to the external eye tracker 20 through the first software processing unit 112 to obtain the eye movement signal, and perform the first foveal processing according to the eye movement signal. In addition, the first software processing unit 112 can be connected to the image signal processing pipeline 12 and the display hardening unit 111 respectively, and perform synchronous alignment processing on the frame synchronization signal of the same frame image in the image signal processing pipeline 12 and the display hardening unit 111.

Specifically, in response to the eye movement signal indicating the user's eye deflection angle, line of sight direction, gaze point coordinates and other information obtained from the eye tracker 20, the grid matrix generation module 13 or the first software processing unit 112 configured with the grid matrix generation program can first determine the user's gaze point area data according to the eye movement signal, and then generate the first grid matrix weight map data according to the gaze point area data. Here, the gaze point area data indicates the coordinate position of the user's gaze point on the image and/or one or more pixels around it. The first grid matrix weight map data respectively indicates the user's attention to multiple areas of the image.

Please further refer to Figure 3, which shows a schematic diagram of a grid matrix weight graph provided according to some embodiments of the present invention.

As shown in FIG3 , in the grid matrix weight map, the user's gaze point may have the highest weight (e.g., 3), multiple neighboring pixels near it may have higher weights (e.g., 2), and multiple edge pixels far from the user's gaze point may have lower weights (e.g., 1). The weights of each pixel may decrease evenly or unevenly from the gaze point. The weights of pixels close to the gaze point are higher, and their grids are relatively dense. On the contrary, the weights of pixels far from the gaze point are lower, and their grids are relatively sparse. In this way, the display hardening unit 111 may perform the first foveal processing on the local first image according to the first grid matrix weight map data. For example, all pixels with a weight of 3 are processed one by one. For another example, one of nearly N pixels with a weight of 2 is processed, and another N-1 pixels are compensated by interpolation, where N is a multiple of 1 or 4. For another example, one of nearly 4N pixels with a weight of 1 is processed, and another 4N-1 pixels are compensated by interpolation, where N is a multiple of 1 or 4. By performing the first foveal processing on the local first image, the present invention can perform the first foveal processing on the local first image in the display pipeline 11 to reduce the amount of data of the first image, thereby improving the image processing quality and user experience while ensuring the central imaging effect of the user's gaze area, and improving the hardware resource utilization of the image processing system 10.

In addition, the first software processing unit 112 can also generate a frame synchronization signal based on the acquired eye movement signal, the determined gaze point area data and/or the generated first grid matrix weight map data, and transmit the frame synchronization signal to the image signal processing pipeline 12, so that the image signal processing pipeline 12 can perform a second foveal processing on the local second image based on the frame synchronization signal. Taking the synchronous alignment of the eye movement signal as an example, the first software processing unit 112 can add a timestamp to the acquired eye movement signal to generate a frame synchronization signal, and transmit the eye movement signal with the timestamp added to the image signal processing pipeline 12.

As shown in FIG1 , the image signal processing pipeline 12 may be preferably configured with an image processing hardening unit 121 and a second software processing unit 122. The image processing hardening unit 121 is configured with one or more transistor-level hardening calculation circuits and/or memories for hardening calculation processing and storage of input signals. The second software processing unit 122 is selected from at least one of a digital signal processor (DSP), a microcontroller (MCU), and a microprocessor, and is configured with auto exposure (AE), auto white balance (AWB), auto focus (AF), anti flicker, auto lens shading correction (ALC), and the like. At least one software program such as ISP (5A module) and grid matrix generation module is used to generate second grid matrix weight map data according to the input frame synchronization signal, and/or set the grid of image information involved in statistical automatic exposure and/or automatic white balance, adjust the algorithm logic of application statistical information, etc. for the input signals and/or data such as the configuration registers and data processing parameters of each hardened calculation circuit and/or memory in the image processing hardening unit 121, and perform software calculation and/or processing for the foveal characteristics of the 5A module.

Specifically, the second software processing unit 122 can be connected to the first software processing unit 112 of the display pipeline 11, and obtains frame synchronization signals such as eye movement signals with time stamps and/or foveal data such as first grid matrix weight map data with time stamps through the first software processing unit 112. Afterwards, the second software processing unit 122 can generate second grid matrix weight map data according to the eye movement signal as described above, and perform second foveal processing based on at least one software program (e.g., 5A module) on the second image of the local corresponding frame according to the time stamp and the generated second grid matrix weight map data. In some embodiments, the resolution and weight of the second grid matrix weight map data generated by the second software processing unit 122 are set according to parameters such as the resolution of the second image, and may be different from the first grid matrix weight map data generated by the first software processing unit 112, thereby further adapting to the processing requirements of the second image and improving the image processing quality of the image signal processing pipeline 12.

In addition, the image processing hardening unit 121 may also be directly connected to the first software processing unit 112 of the display pipeline 11, or connected to the first software processing unit 112 via the second software processing unit 122, so as to directly or indirectly obtain a frame synchronization signal via the display pipeline 11. Here, the frame synchronization signal may be the first grid matrix weight map data generated by the first software processing unit 112, or may be the second grid matrix weight map data generated by the second software processing unit 122. Each hardening calculation circuit in the image processing hardening unit 121 can perform a second foveal processing based on the hardening calculation circuit according to the obtained first and/or second grid matrix weight map data to reduce the data volume of the second image, thereby reducing the power consumption and delay of the image processing system while ensuring the central imaging effect of the user's gaze area, and improving the hardware resource utilization of the image processing system 10.

Furthermore, by using the first grid matrix weight map data as the frame synchronization signal, the image processing hardening unit 121 can directly use the obtained first grid matrix weight map data to perform the second fovea processing, thereby further saving software computing power, saving hardware resources of the hardened computing circuit, and improving the real-time performance of image processing. In particular, in the embodiment of mixed reality (MR) display, by using the same grid matrix weight map data to synchronously perform fovea processing of the virtual rendering image (i.e., the first image) and the real scene image (i.e., the second image), the present invention can effectively improve the adaptability of the virtual object and the ISP real scene, and ensure that the resolution transition of the two is similar, thereby achieving a natural transition between the virtual rendering image and the real scene image.

Please further refer to FIG. 4, which shows a schematic diagram of the architecture of an image signal processing pipeline provided according to some embodiments of the present invention.

In the embodiment shown in FIG. 4 , the image processing hardening unit 121 may be preferably configured with one or more transistor-level hardening calculation circuits such as defective pixel compensation (DPC), demosaic, black level correction (BLC), lens shading correction (LSC), noise reduction (NR), sharpening (SHP), color correction matrix (CCM), gamma correction, and color space conversion (CSC). In addition, the second software processing unit 122 and/or at least one hardening calculation circuit may be respectively configured with independent switches. Afterwards, in response to the second image input locally, the image signal processing pipeline 12 can obtain the switch signal through the image processing hardening unit 121 and the software processing unit 122 respectively, and determine the hardening calculation circuit and/or software processing unit 122 that needs to perform the second fovea processing according to the switch signal. Afterwards, if the switch signal of the software processing unit 122 and/or any one or more hardening calculation circuits is on, the image signal processing pipeline 12 can determine that it needs to perform the second fovea processing, and thus transmit the frame synchronization signal to it, so that the at least one hardening calculation circuit and/or software processing unit 122 can independently perform the second fovea processing. On the contrary, if the switch signal of the software processing unit 122 and/or any one or more hardened computing circuits is OFF, the image signal processing pipeline 12 can determine that it does not need to perform the second fovea processing, thereby skipping the operation of the second fovea processing to provide a personalized fovea processing function.

In some embodiments, the switch signal can be pre-set according to the specific function of the image processing system 10 before the image processing system 10 leaves the factory or is installed and operated. In this way, the image processing system 10 can fixedly control one or more hardening calculation circuits in the image processing hardening unit 121 according to the preset or customized processing method to perform the preset or customized second fovea processing on the local second image.

Furthermore, in some preferred embodiments, the above-mentioned switch signal can also be set frame by frame by the second software processing unit 122. Specifically, in the XR display process, the image signal processing pipeline 12 can extract the image features of the second image of the current frame frame by frame through the second software processing unit 122, using a pre-trained artificial intelligence (AI) model to determine whether it needs to perform a foveal processing operation, as well as specific information such as a grid matrix to be used. Afterwards, the image signal processing pipeline 12 can configure the switch signals of each hardening calculation circuit in real time according to the judgment result that the second image of the current frame needs to perform a foveal processing operation, and synchronously obtain and/or generate specific information related to the foveal processing operation such as a grid matrix. In this way, each hardened computing circuit will perform personalized fovea processing on the second image of the current frame according to the switch signal configured in real time by the second software processing unit 122. Similarly, the image signal processing pipeline 12 can control one or more hardened computing circuits in the image processing hardening unit 121 frame by frame to dynamically perform personalized second fovea processing on the second image of each frame.

Those skilled in the art can understand that the display pipeline 11 and the image signal processing pipeline 12 shown in FIG1 are respectively configured with independent software computing units 112 and 122. This is just a non-restrictive implementation method provided by the present invention, which is intended to clearly demonstrate the main concept of the present invention and provide a specific solution that is convenient for public implementation, rather than to limit the scope of protection of the present invention.

Optionally, in other embodiments, the display pipeline and the image signal processing pipeline may be configured with different hardened units respectively, and share the same software processing unit. In this way, the software processing unit may be connected to the display hardened unit and the image processing hardened unit respectively, and frame synchronization signals such as eye movement signals, grid matrix weight map data, and/or foveal data may be synchronously transmitted between the display hardened unit and the image processing hardened unit to realize data joint interaction and reuse between the display pipeline and the image signal processing pipeline. The specific working method of the software processing unit is similar to the first software processing unit and the second software processing unit mentioned above, and will not be repeated here.

In addition, please continue to refer to FIG. 1. In some embodiments of mixed reality (MR) display, the display pipeline 11 is also connected to the image rendering module 30 such as the central processing unit (CPU) and the graphics processing unit (GPU) of the mixed reality display device, and obtains the virtual rendering image through these image rendering modules 30, and performs the first foveal processing on the virtual rendering image according to the eye movement signal. In addition, the image signal processing pipeline 12 is also connected to the image acquisition device such as the binocular camera 40, obtains the real scene image through the binocular camera 40, and performs the second foveal processing on the real scene image according to the frame synchronization signal provided by the display pipeline 11. Here, since the resolution of the virtual rendering image and the real scene image is usually quite different, the first grid matrix weight map for the first fovea processing and the second grid matrix weight map for the second fovea processing may also be different and involve a relatively large resolution difference.

Furthermore, after the second fovea processing, the image signal processing pipeline 12 can also add a timestamp to the real scene image after the second fovea processing, and synchronously transmit it from the image signal processing pipeline 12 to the display pipeline 11 for display pipeline 11 to perform layer stacking and output it to the subsequent module 14 for mixed reality output display. In this way, the present invention can further reduce the amount of real scene image data transmitted between pipelines to save hardware computing power.

In addition, the display pipeline 11 can find the virtual rendering image of the corresponding frame according to the timestamp of the real scene image, and perform layer blending and mixed reality output display on it with the real scene image. Here, the virtual rendering image can be a compressed image that has undergone the first fovea processing. After completing the layer blending, the display pipeline 11 can directly send the blended image to the subsequent module 14 for mixed reality output display.

Optionally, in other embodiments, the virtual rendering image may also be an original virtual rendering image that has not been subjected to the first foveal processing. In this way, after completing the layer blending, the display pipeline 11 may perform the first foveal processing on the blended image according to the user's eye movement signal, and then send it to the subsequent module 14 for mixed reality output display.

In summary, the above-mentioned image processing system, image processing method, extended reality display device and computer-readable storage medium provided by the present invention can comprehensively improve the image quality, power consumption, real-time performance and user experience of image processing through the processing architecture and processing flow of data joint interaction and reuse of the display pipeline and the image signal processing pipeline, thereby comprehensively improving the current shortage of XR high frame rate display computing power.

Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and/or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but can be understood by those skilled in the art.

Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, code elements, and code chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

The previous description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variants without departing from the spirit or scope of this disclosure. Thus, this disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

10: Image processing system

11: Display pipeline

12: Image signal processing pipeline

13: Grid matrix generation module

14: Subsequent modules

20: Eye tracker

30: Image rendering module

40: Binocular camera

111: Display hardening unit

112: First software processing unit

121: Image processing hardening unit

122: Second software processing unit

Claims (14)

An image processing system based on the foveal principle includes a display pipeline and an image signal processing pipeline, wherein the display pipeline obtains the user's eye movement signal and performs a first foveal processing according to the eye movement signal, and is characterized in that the display pipeline also generates a frame synchronization signal according to the eye movement signal, and transmits the frame synchronization signal to the image signal processing pipeline, so that the image signal processing pipeline performs a second foveal processing according to the frame synchronization signal; wherein the frame synchronization signal includes the eye movement signal, gaze point area data and/or grid matrix weight map data, wherein the gaze point area data indicates the coordinate position of the user's gaze point on the image and/or its surrounding pixels, and the grid matrix weight map data indicates the user's attention to multiple areas of the image. The image processing system as described in claim 1 is characterized in that the display pipeline is configured to: determine the user's gaze point area data according to the eye movement signal; generate the grid matrix weight map data according to the gaze point area data; perform the first foveal processing according to the grid matrix weight map data; and generate the frame synchronization signal according to the eye movement signal, the gaze point area data and/or the grid matrix weight map data. The image processing system as claimed in claim 1 is characterized in that the display pipeline includes a display hardening unit and a first software processing unit, wherein the display hardening unit is connected to an eye tracker via the first software processing unit to obtain the eye movement signal, and performs the first foveal processing according to the eye movement signal, and the first software processing unit is connected to the image signal processing pipeline and the display hardening unit, and synchronously aligns the frame synchronization signals of the same frame image in the image signal processing pipeline and the display hardening unit. The image processing system as claimed in claim 1 is characterized in that the image signal processing pipeline includes an image processing hardened unit and a second software processing unit, wherein the image processing hardened unit is connected to the display pipeline and obtains the frame synchronization signal through the display pipeline to perform a second fovea processing based on at least one hardened computing circuit according to the frame synchronization signal, and the second software processing unit is connected to the display pipeline and obtains the frame synchronization signal through the display pipeline to perform a second fovea processing based on at least one software program according to the frame synchronization signal. The image processing system as described in claim 4 is characterized in that a plurality of the hardening calculation circuits are configured in the image processing hardening unit, wherein at least one of the hardening calculation circuits is configured with an independent switch, and the hardening calculation circuit performs independent second fovea processing according to the switch signal of the corresponding switch. The image processing system as described in claim 5 is characterized in that the second software processing unit extracts the image features of the current frame image frame by frame, and configures the switch signals of each of the switches in real time according to the image features to control one or more of the hardening calculation circuits in the image processing hardening unit frame by frame to perform the second foveal processing, or the switches of each of the hardening calculation circuits in the image processing hardening unit are pre-set to corresponding switch states based on the image processing function of the image processing system to fixedly control one or more of the hardening calculation circuits in the image processing hardening unit to perform the second foveal processing. The image processing system as described in claim 1 is characterized in that the display pipeline is also connected to the image rendering module, and the virtual rendered image is obtained through the image rendering module, and the first foveal processing is performed on the virtual rendered image according to the eye movement signal. The image signal processing pipeline is also connected to the camera, and the real scene image is obtained through the camera, and the second foveal processing is performed on the real scene image according to the frame synchronization signal. The image signal processing pipeline also transmits the real scene image that has undergone the second foveal processing to the display pipeline to perform layer blending with the virtual rendered image that has undergone or has not undergone the first foveal processing. An image processing method based on the foveal principle is executed by a processor of an image processing system, and is characterized in that it includes the following steps: obtaining an eye movement signal of a user; transmitting the eye movement signal to a display pipeline of the image processing system to perform a first foveal processing and generate a frame synchronization signal; and transmitting the frame synchronization signal to an image signal processing pipeline of the image processing system for the image processing system to generate a frame synchronization signal. The signal processing pipeline performs the second foveal processing according to the frame synchronization signal; Wherein, the frame synchronization signal includes the eye movement signal, the gaze point area data and/or the grid matrix weight map data, wherein the gaze point area data indicates the coordinate position of the user's gaze point on the image and/or its surrounding pixels, and the grid matrix weight map data indicates the user's attention to multiple areas of the image. The image processing method as described in claim 8 is characterized in that the step of transmitting the eye movement signal to the display pipeline of the image processing system to perform the first foveal processing and generate the frame synchronization signal includes: determining the gaze point of the user according to the eye movement signal; generating grid matrix weight map data according to the gaze point; performing the first foveal processing according to the grid matrix weight map data; and generating the frame synchronization signal according to the eye movement signal and/or the grid matrix weight map data. The image processing method as described in claim 8 is characterized in that the image signal processing pipeline includes an image processing hardening unit and a software processing unit, the image processing hardening unit is configured with at least one hardened computing circuit, and the step of transmitting the frame synchronization signal to the image signal processing pipeline of the image processing system so that the image signal processing pipeline performs the second fovea processing according to the frame synchronization signal comprises: transmitting the frame synchronization signal to the image signal processing pipeline of the image processing system through the image processing unit; The hardening unit and the software processing unit obtain a switch signal; according to the switch signal, determine the hardening calculation circuit and/or software processing unit that needs to perform the second fovea processing; and transmit the frame synchronization signal to at least one hardening calculation circuit and/or software processing unit that needs to perform the second fovea processing, so that the at least one hardening calculation circuit and/or software processing unit independently performs the second fovea processing. The image processing method as claimed in claim 10 is characterized in that a plurality of the hardening calculation circuits are configured in the image processing hardening unit, and the step of determining the hardening calculation circuit and/or software processing unit that needs to perform the second fovea processing according to the switch signal comprises: extracting the image features of the current frame image frame by frame through the software processing unit, and configuring the switch signals of each of the hardening calculation circuits in real time according to the image features, so as to perform the second fovea processing one by one. The frame controls one or more of the hardening calculation circuits in the image processing hardening unit to perform the second foveal processing; or based on the image processing function of the image processing system, the switch state of each of the hardening calculation circuits in the image processing hardening unit is pre-set, and according to the switch state of each of the hardening calculation circuits, one or more of the hardening calculation circuits in the image processing hardening unit is fixedly controlled to perform the second foveal processing. The image processing method as described in claim 8 is characterized in that the display pipeline is also connected to an image rendering module, a virtual rendering image is obtained through the image rendering module, and the first foveal processing is performed on the virtual rendering image according to the eye movement signal, the image signal processing pipeline is also connected to a camera, a real scene image is obtained through the camera, and the real scene image is subjected to the second foveal processing according to the frame synchronization signal, and after the second foveal processing is performed, the image processing method further includes the following steps: the real scene image subjected to the second foveal processing is transmitted from the image signal processing pipeline to the display pipeline to be layered with the virtual rendering image subjected to or not subjected to the first foveal processing. An extended reality display device, characterized in that it includes an image processing system as described in any one of claims 1 to 7. A computer-readable storage medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, an image processing method as described in any one of claims 8 to 12 is implemented.