WO2022227753A1 - Display system and display method for binocular distortion correction, and vehicle-mounted system - Google Patents

Abstract

Provided in the present application are a display system and display method for binocular distortion correction, and a vehicle-mounted system, which can be applied to an intelligent vehicle, a networked vehicle and an autonomous vehicle. The display system comprises a picture generation unit, a binocular parallax generation unit and an optical lens group, wherein the picture generation unit is used for generating a left-eye pre-distorted picture and a right-eye pre-distorted picture, and for interleaving pixels in the left-eye pre-distorted picture and pixels in the right-eye pre-distorted picture to generate a pixel-interleaved composite picture; and the binocular parallax generation unit makes the pixel-interleaved composite picture be pushed to the two eyes of an observer by means of the optical lens group, the pixels of the left-eye pre-distorted picture in the pixel-interleaved composite picture being pushed to the left eye of the observer, and the pixels of the right-eye pre-distorted picture in the pixel-interleaved composite picture being pushed to the right eye of the observer. By means of the solution of the present application, two eyes can simultaneously receive undistorted pictures which then overlap, such that the overall visual perception can be improved.

Description

Display system, display method and vehicle-mounted system for binocular distortion correction

This application claims the priority of the Chinese patent application filed on April 28, 2021 with the State Intellectual Property Office of China, the application number is 202110468890.9, and the application name is "display system, display method and vehicle-mounted system for binocular distortion correction", which The entire contents of this application are incorporated by reference.

technical field

The present application relates to the field of image processing, and more particularly, to a display system, a display method and a vehicle-mounted system for binocular distortion correction.

Background technique

Known visual display systems generally include a picture generation unit (PGU), a projector and an optical set. Among them, the projector receives the image supplied from the image generator and projects it, and then the light path passes through the precisely designed optical lens group, and finally the image is reflected to the observer's visual range - the eye box range.

However, due to the design limitations of the components in the display system, such as the irregular surface of the mirror or the deviation of the assembly process, the imaging results in different degrees of distortion; and the observer's binocular is located in different positions of the eye box, which leads to binocular vision. At the same time, images with different distortions are received without overlapping, which ultimately greatly reduces the overall visual experience.

Therefore, how to improve the overall visual experience is an urgent problem to be solved.

SUMMARY OF THE INVENTION

The present application provides a display system, a display method and a vehicle-mounted system for binocular distortion correction, which can improve the overall visual experience.

In a first aspect, a display system for binocular distortion correction is provided, including: an image generator, a binocular parallax generator, and an optical lens group; the image generator is used to generate a left-eye pre-distortion image and a right-eye pre-distortion image , and the pixels in the left-eye pre-distorted image are interleaved with the pixels in the right-eye pre-distorted image to generate a composite image of pixel interleaving; the binocular parallax generator makes the composite image of pixel interleaving pushed to the observer through the optical lens group , in which pixels of the left-eye pre-distorted image in the pixel-interleaved composite image are pushed to the observer's left eye, and pixels of the right-eye pre-distorted image in the pixel-interleaved composite image are pushed to the observer's right eye.

The display system for binocular distortion correction provided by the embodiment of the present application includes an image generator, a binocular parallax generator, and an optical lens group. Among them, the image generator can generate a left-eye pre-distorted image and a right-eye pre-distorted image, and can interleave the pixels in the left-eye pre-distorted image with the pixels in the right-eye pre-distorted image to generate a pixel-interleaved composite image; the binocular disparity generator can make The pixels of the left eye pre-distorted image and the pixels of the right eye pre-distorted image in the pixel interleaving synthesis image are respectively pushed to the left eye and the right eye of the observer through the optical lens group. The solution of the present application enables binoculars to receive images without distortion at the same time and overlap, thereby improving the overall visual experience.

With reference to the first aspect, in some implementations of the first aspect, the image generator is further configured to overlap the left-eye pre-distorted image and the right-eye pre-distorted image; and determine an envelope range, where the envelope range covers the left eye The pre-distorted image and the effective pixels in the right-eye pre-distorted image; the pixels in the left-eye pre-distorted image and the right-eye pre-distorted image are interleaved within the envelope range to generate a pixel interleaving composite image.

With reference to the first aspect, in some implementations of the first aspect, the display system further includes: a distortion corrector; the distortion corrector is used to obtain a left-eye distortion model and a right-eye distortion model, where the left-eye distortion model is based on the target source image The sample and the uncorrected left eye distortion imaging sample are generated, the right eye distortion model is generated according to the target source image sample and the uncorrected right eye distortion imaging sample; the image generator is also used to obtain the target source image; according to the The target source image, the left-eye distortion model, and the right-eye distortion model generate the left-eye pre-distortion image and the right-eye pre-distortion image.

With reference to the first aspect, in some implementations of the first aspect, the target source image is a source image captured at a target perspective; the image generator is further configured to generate the left eye according to the target source image and the left eye distortion model A pre-distorted image, the right-eye pre-distorted image is generated according to the target source image and the right-eye distortion model.

Optionally, the target viewing angle may be a viewing angle between the left eye and the right eye, or may be another viewing angle, which is not limited in this application.

In this embodiment of the present application, the target source image is the source image captured at the target viewing angle, and pre-distortion processing is performed according to the same source image (ie, the target source image) to obtain the left-eye pre-distortion image and the right-eye pre-distortion image, so that the binocular location The observed imaging is a two-dimensional 2D imaging without distortion.

With reference to the first aspect, in some implementations of the first aspect, the target source image includes a left eye perspective target source image and a right eye perspective target source image; the image generator is further configured to, according to the left eye perspective target source image and the The left-eye distortion model generates the left-eye pre-distortion image, and the right-eye pre-distortion image is generated according to the right-eye perspective target source image and the right-eye distortion model.

In the embodiment of the present application, the target source image includes a left eye perspective target source image and a right eye perspective target source image, and images from different perspectives (ie, the left eye perspective target source image and the right eye perspective target source image) are respectively subjected to pre-distortion processing to obtain the left eye perspective. The pre-distorted image and the right-eye pre-distorted image, so that the image observed in the binocular is an undistorted three-dimensional 3D image.

With reference to the first aspect, in some implementations of the first aspect, the display system further includes: an eye tracker; the eye tracker is used to detect the binocular coordinates of the observer; the distortion corrector is also used to , and the left-eye distortion model and the right-eye distortion model are obtained according to the observer's binocular coordinates.

In the embodiment of the present application, the display system may further include: an eye tracker, which can be used to detect the observer's binocular coordinates, so as to obtain the left-eye distortion model and the right-eye distortion model according to the binocular coordinate positions, so that the left eye is distorted The model and the right eye distortion model can be updated in real time according to the binocular coordinates, so that the observer can always see the distortion-free image in different positions in the eye box, which can further improve the overall visual experience.

With reference to the first aspect, in some implementations of the first aspect, the uncorrected left-eye distortion imaging sample and the uncorrected right-eye distortion imaging sample are based on the observer's binocular coordinates and the final imaging plane. The target position is obtained from the spatial transformation relationship between the coordinates of the target position, where the target position is the intersection of the extension line of the left eye line of sight and the right eye line of sight.

With reference to the first aspect, in some implementations of the first aspect, the optical mirror group includes a flat mirror and a free-form mirror.

In the embodiment of the present application, the optical lens group may further include a self-owned curved mirror, and the self-owned curved mirror has the functions of imaging magnification and enlarging the scope of the eye box, so that the observer can enjoy a better viewing experience.

With reference to the first aspect, in some implementations of the first aspect, the binocular parallax generator is a parallax barrier or a directional light source component.

With reference to the first aspect, in some implementations of the first aspect, the parallax barrier is a diaphragm or a lenticular lens.

With reference to the first aspect, in some implementations of the first aspect, the display system is an office entertainment display system or a vehicle head-up display system.

In a second aspect, a display method for binocular distortion correction is provided, including: generating a left-eye pre-distortion image and a right-eye pre-distortion image; Interleaving generates a pixel interleaving composite image; under the action of the binocular parallax generator, the pixel interweaving composite image is pushed to the observer's binocular through the optical lens group, wherein the pixels in the pixel interleaving composite image are the pixels of the left-eye pre-distorted image. Pushed to the observer's left eye, the pixels of the right-eye predistorted image in the pixel-interleaved composite image are pushed to the observer's right eye.

With reference to the second aspect, in some implementations of the second aspect, the interleaving of the pixels in the pre-distorted image for the left eye and the pixels in the pre-distorted image for the right eye to generate a composite pixel interleaving image includes: the pre-distorted image for the left eye. overlap with the right-eye pre-distorted image; determine an envelope range, the envelope range covers the effective pixels in the left-eye pre-distorted image and the right-eye pre-distorted image; the pixels in the left-eye pre-distorted image and the right-eye pre-distorted image The pixels of are interleaved within the envelope range to generate a pixel interleaved composite image.

With reference to the second aspect, in some implementations of the second aspect, the method further includes: acquiring a left-eye distortion model and a right-eye distortion model, where the left-eye distortion model is generated according to a target source image sample and an uncorrected left-eye distortion image sample , the right-eye distortion model is generated according to the target source image sample and the uncorrected right-eye distortion image sample; the generating the left-eye pre-distortion image and the right-eye pre-distortion image includes: acquiring the target source image; The distortion model and the right-eye distortion model generate the left-eye pre-distortion image and the right-eye pre-distortion image.

In combination with the second aspect, in some implementations of the second aspect, the target source image is a source image captured at a target perspective; the left-eye pre-distortion is generated according to the target source image, the left-eye distortion model and the right-eye distortion model The image and the right-eye pre-distortion image include: generating the left-eye pre-distortion image according to the target source image and the left-eye distortion model, and generating the right-eye pre-distortion image according to the target source image and the right-eye distortion model.

With reference to the second aspect, in some implementations of the second aspect, the target source image includes a left eye perspective target source image and a right eye perspective target source image; the target source image is generated according to the target source image, the left eye distortion model and the right eye distortion model The left-eye pre-distorted image and the right-eye pre-distorted image include: generating the left-eye pre-distorted image according to the left-eye perspective target source image and the left-eye distortion model, and generating the right-eye pre-distorted image according to the right-eye perspective target source image and the right-eye distortion model .

With reference to the second aspect, in some implementations of the second aspect, the method further includes: detecting the observer's binocular coordinates; the acquiring the left-eye distortion model and the right-eye distortion model includes: acquiring the observer's binocular coordinates according to the The left eye distortion model and the right eye distortion model.

With reference to the second aspect, in some implementations of the second aspect, the uncorrected left-eye distortion imaging sample and the uncorrected right-eye distortion imaging sample are based on the observer's binocular coordinates and the final imaging plane. The target position is obtained from the spatial transformation relationship between the coordinates of the target position, where the target position is the intersection of the extension line of the left eye line of sight and the right eye line of sight.

In combination with the second aspect, in some implementations of the second aspect, the optical mirror group includes a flat mirror and a free-form mirror.

In combination with the second aspect, in some implementations of the second aspect, the binocular parallax generator is a parallax barrier or a directional light source component.

In combination with the second aspect, in some implementations of the second aspect, the parallax barrier is a diaphragm or a lenticular lens.

In a third aspect, a controller is provided, including an input and output interface, a processor and a memory, the processor is used to control the input and output interface to send and receive signals or information, the memory is used to store a computer program, and the processor is used to control the The computer program is called and executed in the memory, so that the controller executes the display method for binocular distortion correction according to the second aspect or any possible implementation manner of the second aspect.

In a fourth aspect, an in-vehicle system is provided, including the display system for binocular distortion correction according to the first aspect or any possible implementation manner of the first aspect.

In a fifth aspect, a desktop display system is provided, including the display system for binocular distortion correction according to the first aspect or any possible implementation manner of the first aspect.

In a sixth aspect, a vehicle is provided, including the display system for binocular distortion correction as in the first aspect or in any possible implementation manner of the first aspect.

In a seventh aspect, a computing device is provided, comprising: at least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory to execute the second The display method for binocular distortion correction in the aspect or any possible implementation manner of the second aspect.

In an eighth aspect, there is provided a computer program product comprising instructions, when the computer program product is run on a computer, the computer program product enables the computer to perform the binocular application in the second aspect or any possible implementation manner of the second aspect. Display method for distortion correction.

In a ninth aspect, a computer-readable storage medium is provided, the computer-readable storage medium is used to store a computer program, and the computer program includes the method for executing the second aspect or any possible implementation manner of the second aspect. The instructions for the display method for binocular distortion correction.

In a tenth aspect, a computer program is provided, the computer program comprising instructions for executing the display method for binocular distortion correction in the second aspect or any possible implementation manner of the second aspect.

In an eleventh aspect, a chip is provided, the chip includes a processor and a data interface, the processor reads an instruction stored in a memory through the data interface, and executes the second aspect or any possible operation of the second aspect. A display method for binocular distortion correction in an implementation.

Optionally, as an implementation manner, the chip may further include a memory, in which instructions are stored, the processor is configured to execute the instructions stored in the memory, and when the instructions are executed, the The processor is configured to execute the display method for binocular distortion correction in the second aspect or any possible implementation manner of the second aspect.

A twelfth aspect provides a system-on-a-chip, the system-on-a-chip includes at least one processor for supporting functions involved in implementing the above second aspect or some implementations of the second aspect, for example, receiving or processing the above The data and/or information involved in the method.

In a possible design, the chip system further includes a memory for storing program instructions and data, and the memory is located inside the processor or outside the processor. The chip system may be composed of chips, or may include chips and other discrete devices.

Description of drawings

FIG. 1 is an example diagram of the principle of performing distortion correction on a single image provided by an embodiment of the present application;

FIG. 2 is an example diagram of a display system for binocular distortion correction provided by an embodiment of the present application;

FIG. 3 is an example diagram of generating a pixel interleaving composite graph provided by an embodiment of the present application;

FIG. 4 is an example diagram of the working principle of a binocular disparity generator provided by an embodiment of the present application;

FIG. 5 is an example diagram of a system architecture provided by an embodiment of the present application;

6 is an example diagram of a system architecture for use in an office entertainment display scenario provided by an embodiment of the present application;

7 is an example diagram of a system architecture for a vehicle head-up display scene provided by an embodiment of the present application;

8 is an example diagram of a binocular distortion correction method provided by an embodiment of the present application;

9 is an example diagram of a display method for binocular distortion correction provided by an embodiment of the present application;

FIG. 10 is an exemplary block diagram of a hardware structure of an apparatus provided by an embodiment of the present application.

Detailed ways

For ease of understanding, the background technology involved in the embodiments of the present application is first introduced in detail.

Known visual display systems generally include a picture generation unit (PGU), a projector and an optical set. Among them, the projector receives the image supplied from the image generator and projects it, and then the light path passes through the precisely designed optical lens group, and finally the image is reflected to the observer's visual range - the eye box range.

However, due to the design limitations of the components in the display system, such as the irregular curved surface of the mirror or the deviation of the assembly process, the imaging results in different degrees of distortion; and because the observer's left and right eyes are located in different positions of the eye box, the two eyes are simultaneously Receive images with different distortions without overlapping. In the end, the overall visual experience is greatly reduced.

In order to solve the image distortion problem caused by the limitations of the above imaging principles, the current mainstream image processing method is to perform distortion correction on the entire image seen by one eye, while the image seen by the other eye is its own distortion. Overlay of effects and predistortion corrections processed for the first eye. However, this correction scheme has poor correction effect on image distortion when the difference of binocular imaging distortion is large.

FIG. 1 is a schematic diagram of a principle example of distortion correction for a single image provided by an embodiment of the present application. As shown in Figure 1, first define I as the original image of the projector, and J as the image received by the observer. Then, a mapping model M from the pixels on the imaging J to the pixels on the original image I can be established through the pixel pairs corresponding to the original image I and the imaging J. So there are:

I=M(J) (1)

If the original image I is an undistorted image and J is an image with distortion, then the mapping model M can be recorded as the distortion model M. Then the specific representation of the distortion model M can be obtained from formula (1). Then, the inverse mapping process is performed on the mapping model M to obtain an inverse mapping model M ^-1 from the original image to the final image.

Finally, the pre-distorted image I' in the projector is calculated according to the mapping model M obtained by formula (1). In practice, if you want to observe undistorted imaging, you can project the pre-distorted image I', and obtain the undistorted final image J' through the inverse mapping model M ^-1 imaging. which is:

J'=M ^-1 (I') (2)

Since the left and right eyes of the observer are located in different positions of the eye box, the two eyes will simultaneously receive images with different distortions without overlapping. To this end, the corresponding distortion models can be obtained for the left and right views respectively, and the left and right pre-distorted images can be prepared according to the distortion models. Its final images are superimposed to enhance the user's visual experience.

However, in the prior art, after obtaining the left and right pre-distorted images, the image generator periodically switches the left and right pre-distorted images at high speed, and then uses the parallax barrier technology or the directional light source technology to cooperate with the time-sharing. The pre-distorted images of the left eye are respectively sent to the left eye of the observer through the optical lens group, and the pre-distorted images of the right eye are respectively sent to the right eye of the observer through the optical lens group.

However, since the frame rate required by the human eye for screen switching needs to reach 60fps or above, it is impossible to perceive the alternation of the left and right images to achieve a nonsensical effect. This means that the above existing solution needs to switch the left and right views within one frame, so the switching speed of the image generator needs to reach 2*60Hz=120Hz or more. The high switching rate requirement will cause a certain computational pressure to the data reading system, especially when dealing with high-definition images.

Based on this, the present application provides a display system for binocular distortion correction. The display system generates a pixel-interleaved composite image by interleaving pixels in a left-eye pre-distorted image with pixels in a right-eye pre-distortion image, and then interleaving the pixels. Under the action of the binocular disparity generator, the composite image is pushed to the binocular in the way of spatial division multiplexing, which can improve the user's visual experience and reduce the calculation pressure of the data reading system.

The technical solutions of the present application will be described below with reference to the accompanying drawings.

FIG. 2 is an example diagram of a display system for binocular distortion correction provided by an embodiment of the present application. Optionally, the display system may be an office entertainment display system or a vehicle head-up display system, which is not limited in this application. As shown in FIG. 2 , the display system 200 includes an image generator 210 , a binocular parallax generator 220 and an optical lens group 230 .

The image generator 210 is used for generating a left eye pre-distorted image and a right eye pre-distorted image.

The image generator 210 is further configured to interleave the pixels in the pre-distorted image for the left eye and the pixels in the pre-distorted image for the right eye to generate a composite image by interleaving.

The binocular disparity generator 220 is used to make the pixel interleaved composite image pushed to the observer's binocular through the optical lens group 230, wherein the pixels of the left eye pre-distorted image in the pixel interleaved composite image are pushed to the observer's left eye, and the pixel The pixels of the right eye predistorted image in the interleaved composite image are pushed to the observer's right eye.

Optionally, the binocular parallax generator is a parallax barrier or a directional light source component. Optionally, the binocular disparity generator can be installed in the image generator or can be independent of the image generator. Optionally, the parallax barrier is a diaphragm or a lenticular lens.

Optionally, the image generator 220 may generate the pixel interleaving composite image through the following process: overlapping the left-eye pre-distorted image and the right-eye pre-distorted image; Effective pixels; the pixels in the pre-distorted image of the left eye and the pixels in the pre-distorted image of the right eye are interleaved within the envelope range to generate a composite image of pixel interleaving.

Optionally, when the left-eye pre-distorted image and the right-eye pre-distorted image are overlapped, it can be based on the center overlap of the left-eye pre-distorted image and the right-eye pre-distorted image; it can also be based on the alignment of the display content of the left-eye pre-distorted image and the right-eye pre-distorted image. ; It can also be overlapped based on other positions, and this application does not specifically limit the overlapped manner. However, for the convenience of description, the center overlap will be taken as an example for description in FIG. 3 below.

Optionally, the pixels in the pre-distorted image for the left eye and the pixels in the pre-distorted image for the right eye may be performed in the manner of LRLR... (wherein L is the pixel in the pre-distorted image for the left eye, and R is the pixel in the pre-distorted image for the right eye). Interleaving can also be performed in the manner of LLRRLLRR..., or in other irregular manners such as LLLLR..., which is not limited in this application, only under the action of the matching binocular disparity generator, The pixels in the left-eye pre-distorted image in the pixel interleaving synthesis image are pushed to the left eye of the observer, and the pixels in the right-eye pre-distorted image in the pixel interleaving synthesis image are pushed to the observer's right eye. For convenience of description, the above-mentioned first interleaving manner will be used as an example for description in FIG. 3 below.

It should be understood that the purpose of defining the envelope range is to ensure that no pixels are lost in the final image due to the misalignment of the shapes of the left and right pre-distorted images. Optionally, the envelope range may be a rectangle or other shapes, which are not limited in this application.

Exemplarily, FIG. 3 is an example diagram of generating a pixel interleaving composite graph provided by an embodiment of the present application. It should be understood that FIG. 3 is only an example, and does not constitute a limitation of the present application. Specifically, the centers of the left eye pre-distorted image 202 and the right eye pre-distorted image 203 are first overlapped. Next, a rectangular envelope range 201 is determined, the rectangular envelope range 201 can cover all effective pixels in the left-eye pre-distorted image 202 and the right-eye pre-distorted image 203 , as shown in (a) of FIG. 3 . Finally, the image generator 210 takes the effective pixels of the left-eye pre-distorted image 202 and the right-eye pre-distorted image 203 respectively in the envelope range 201, and as shown in (b) of FIG. The pixels of the pre-distorted image 202 are interleaved with the pixels of the right-eye pre-distorted image 203 to generate a new composite image 301 of pixel interleaving. After obtaining the pixel interleaving composite image 301, the pixel interleaving composite image 301 is pushed to the observer's binocular under the action of the parallax barrier 302 matching the pixel interleaving composite image 301, wherein the pixel interleaving composite image 301 in the left eye The pixels of the pre-distorted image are pushed to the left eye of the observer, and the pixels of the pre-distorted image of the right eye in the pixel interleaving composite diagram 301 are pushed to the right eye of the observer, as shown in FIG. 4 .

In the embodiment of the present application, a pixel interleaving composite image is generated by interleaving the pixels in the left-eye pre-distorted image with the pixels in the right-eye pre-distortion image, and then, under the action of the binocular disparity generator, in a space-division multiplexing manner, The pixels of the left-eye pre-distorted image and the right-eye pre-distorted image in the pixel-interleaving composite image are respectively pushed to the observer's binocular, so that the overall visual experience can be improved. Moreover, the calculation pressure on the data reading system caused by the high-frequency switching of the left and right pre-distorted images in the time division multiplexing method in the prior art is effectively avoided.

Optionally, the image generator 210 can also generate multiple (more than 2) pre-distorted images at the same time, and interleave the pixels in the multiple pre-distorted images to generate a pixel-interleaved composite image. The present application does not limit multiple pre-distorted images. The way in which the pixels are interleaved. The multiple pre-distorted images may correspond to multiple pre-distorted images from different viewing angles. Then, with the cooperation of the binocular disparity generator 230, the pixels corresponding to the two pre-distorted images in the multiple pre-distorted images are respectively pushed to the observer's binocular. And with the binocular movement of the observer, the two pre-distorted images that can be projected to the binoculars change continuously. For example, the image generator can generate the pre-distorted image 1, the pre-distorted image 2, the pre-distorted image 3 and the pre-distorted image 4 at the same time, and interleave the pixels in the above four pre-distorted images to generate a pixel-interleaved composite image. If the current binocular is at position 1, then with the cooperation of the binocular disparity generator 230, the pixels corresponding to the pre-distorted image 1 and the pre-distorted image 2 in the pixel interleaving composite image are respectively pushed to the observer's binocular; If the observer binocular moves to position 2, with the cooperation of the binocular disparity generator 230, the pixels corresponding to the pre-distorted image 3 and the pre-distorted image 4 in the pixel interleaving composite image are respectively pushed to the observer's binocular eye. However, for the convenience of description, the following descriptions are given by taking the image generator generating the pre-distorted image for the left eye and the pre-distorted image for the right eye as an example.

Optionally, the optical mirror group 230 may include a flat mirror and a free-form surface mirror. For details, please refer to the description of the optical mirror group in the system architectures 500 to 700 below, which will not be repeated here.

Optionally, the display system 200 may further include a distortion corrector, and the distortion corrector may exist in the image generator 210, or may exist independently of the image generator 210, which is not limited in this application. The distortion corrector can be used to obtain a left eye distortion model and a right eye distortion model. Among them, the left eye distortion model is generated from the target source image sample and the uncorrected left eye distortion image sample, and the right eye distortion model is generated from the target source image sample and the uncorrected right eye distortion image sample.

It should be understood that the uncorrected left eye distortion imaging sample and the uncorrected right eye distortion imaging sample can be collectively referred to as the uncorrected distortion imaging sample, and the uncorrected left eye distortion imaging sample is the uncorrected image observed at the left eye coordinate position. The corrected distorted image sample, the uncorrected right eye distorted image sample is the uncorrected distorted image sample observed at the right eye coordinate position. The left eye coordinate position and the right eye coordinate position can also be understood as different coordinate positions in the eye box.

Optionally, the distortion corrector obtains the left-eye distortion model and the right-eye distortion model by directly reading the left-eye distortion model and the right-eye distortion model from the display system. It should be understood that in this case, the display system needs to generate the left distortion model and right distortion model corresponding to the binocular in advance before leaving the factory or use, and store them in the system, so that in the actual use process, the model can be directly read. Pick. Specifically, it is necessary to generate a left-eye distortion model according to the target source image sample and the uncorrected left-eye distortion image sample in advance, and generate a right-eye distortion model according to the target source image sample and the uncorrected right-eye distortion image sample (implemented by formula (1)).

Optionally, the distortion model obtained by the distortion corrector may also be obtained by obtaining the distortion model generated by itself. It should be understood that in this case, the distortion corrector can first obtain the target source image sample, the uncorrected left eye distortion imaging sample and the uncorrected right eye distortion imaging sample, and then the distortion corrector can obtain the target source image sample corresponding to the left eye. The image sample and the uncorrected left-eye distortion imaging sample generate the left-eye distortion model, and the right-eye distortion model is generated according to the target source image sample corresponding to the right-eye and the uncorrected right-eye distortion image sample (implemented by formula (1)).

Optionally, after acquiring the left-eye distortion model and the right-eye distortion model, the image generator 210 can acquire the target source image, and generate the left-eye pre-distortion image and the right-eye pre-distortion image according to the target source image, the left-eye distortion model, and the right-eye distortion model. .

It should be understood that the target source image is the source image that the display system needs to display during actual use, that is, the image that needs to be displayed for the user.

Optionally, the target source image may be a source image captured at the target viewing angle. The image generator 210 may generate a left-eye pre-distorted image according to the target source image and the left-eye distortion model, and generate a right-eye pre-distortion image according to the target source image and the right-eye distortion model (achieved by formula (2)).

Optionally, the target viewing angle may be a viewing angle between the left eye and the right eye, or may be another viewing angle, which is not limited in this application. In the embodiment of the present application, the target source image is the source image captured at the target viewing angle, and in this case, the left eye predistorted image and the right eye predistorted image are obtained by performing pre-distortion processing according to the same source image (ie, the target source image). , so that the image observed at the binocular is a 2D image without distortion.

Optionally, the target source image may include a left eye perspective target source image and a right eye perspective target source image. The image generator 210 may generate a left-eye pre-distorted image according to the left-eye view target source image and the left-eye distortion model, and generate a right-eye pre-distortion image according to the right-eye view target source image and the right-eye distortion model (implemented by formula (2)).

Optionally, the target source image from the left eye perspective and the target source image from the right eye perspective may be obtained by photographing the target source image respectively according to the left eye perspective and the right eye perspective; or the target source image may be photographed according to one of the left eye perspective or the right eye perspective. The left eye perspective or the right eye perspective is obtained by shooting, and then the view from the obtained perspective is combined with the image processing algorithm to generate a view from another perspective; it can also be obtained by shooting a view from other perspectives, and then according to the captured view. and the view from the right eye perspective, the present application does not limit the acquisition methods of the left eye perspective target source image and the right eye perspective target source image.

In the embodiment of the present application, images from different perspectives (ie, the target source image from the left eye perspective and the target source image from the right eye perspective) are respectively pre-distorted to obtain the left eye pre-distorted image and the right eye pre-distorted image, so that the images observed in the binocular place are obtained. Imaging is distortion-free 3D imaging.

Because in practical applications, the binocular position of the observer changes dynamically, rather than being fixed. If the fixed left-eye distortion model and right-eye distortion model are used, it will not be able to meet the requirement of the observer to always see an undistorted image in the eye box. Thus, optionally, the display system 200 may further include: an eye tracker. Eye trackers can be used to detect the binocular coordinates of the observer.

At this time, the distortion corrector can also be used to obtain the left-eye distortion model and the right-eye distortion model according to the binocular coordinates of the observer.

Optionally, the distortion corrector may directly read the left-eye distortion model and the right-eye distortion model corresponding to the current left-eye and right-eye coordinates from the parameter table in the system according to the current binocular coordinate information. It should be understood that in this case, the display system needs to generate the distortion models corresponding to different coordinate positions of the eye box in advance before leaving the factory or use, and store them in the system, so that in the actual use process, it can be read directly according to the coordinate position of the left eye. Take the corresponding left eye distortion model and read the corresponding right eye distortion model according to the coordinate position of the right eye. This means that the display system needs to generate distortion models corresponding to different coordinate positions according to the target source image samples and uncorrected distortion image samples corresponding to different coordinate positions of the eye box in advance before leaving the factory or using the formula (implemented by formula (1)) .

Optionally, the distortion model obtained by the distortion corrector may also be obtained by obtaining the distortion model generated by itself. It should be understood that in this case, the distortion corrector can first obtain the target source image sample corresponding to the current coordinate position of the left eye, the uncorrected left eye distortion image sample, and the target source image sample corresponding to the current coordinate position of the right eye. and the uncorrected right eye distortion imaging sample, and then the distortion corrector generates a left eye distortion model according to the target source image sample corresponding to the current coordinate position of the left eye and the uncorrected left eye distortion imaging sample, according to the current coordinate position of the right eye. The target source image samples and the uncorrected right-eye distortion image samples generate a right-eye distortion model (implemented using formula (1)). It should be understood that in this case, the system needs to save the corresponding target source image samples and uncorrected distorted image samples in different coordinate positions of the eye box in advance; Each uncorrected distorted image of the image sample at the location is obtained, and then in the actual operation, according to the current coordinate position of the binocular, the camera shooting image corresponding to the current coordinate position of the binocular is obtained respectively, and then combined with the current coordinate of the binocular. The spatial transformation relationship between the position and the coordinates of the target position on the final imaging plane transforms the camera shot map into the desired sample map as described above.

It should be understood that the uncorrected distorted imaging samples can be obtained according to the spatial transformation relationship between the different coordinate positions of the eye box and the coordinates of the target position on the final imaging plane, where the target position is the intersection of the extension line of the left eye line of sight and the right eye line of sight. . The way of determining the spatial transformation relationship can be found in the following description. It should be understood that the spatial transformation relationship in the embodiments of the present application may be generated by a distortion corrector, or may be generated by an eye tracker, or may be generated in advance by the system and stored in the system, so that it can be directly obtained from the system during use , which is not limited in this application.

The following briefly introduces the acquisition methods of the target source image sample, the uncorrected left-eye distortion imaging sample, and the uncorrected right-eye distortion imaging sample.

First of all, it should be understood that it can be seen from formula (1) that the acquisition of the distortion model M depends on the comparison between the original image I without distortion and the image J with distortion. Wherein, the undistorted original image I is the undistorted original image projected by the projector, which is denoted as the target source image sample in this application; the distorted image J is the distorted image received at the target position of the eye box, Correspondingly, this application records it as an uncorrected distorted image sample, the uncorrected left-eye distortion image sample received at the left eye coordinate position of the specific eye box is an uncorrected image sample received at the right eye coordinate position of the eye box. The right eye distorted image sample.

Then, in actual operation, after the imaging position of the uncorrected distorted imaging sample is determined, the uncorrected distorted imaging sample needs to be acquired. It should be understood that the present application does not limit the manner of acquiring the uncorrected distorted image samples. Optionally, it can be obtained by taking pictures with a camera. Specifically, a camera can be used to acquire the uncorrected distorted image samples at the target position of the eye box. However, there is a geometric relationship between the projection mapping of the world's three-dimensional information (including image information) captured by the camera to the two-dimensional imaging plane. Therefore, the uncorrected distorted image samples cannot be obtained directly by placing the camera at the target position of the eye box and shooting separately. And because of this geometric relationship (ie, the spatial transformation relationship), it can be obtained through camera calibration. Therefore, the above-mentioned uncorrected distorted image sample can be obtained through the geometric relationship and the image captured by the camera. Specifically, the geometric relationship between the coordinate position of the left eye and the target position on the final imaging plane and the image captured by the camera at the position of the left eye can be obtained. The uncorrected left-eye distortion image sample is obtained from the image, and the uncorrected right-eye distortion image sample is obtained through the geometric relationship between the right-eye coordinate position and the target position on the final imaging plane and the image captured by the camera at the right-eye position.

Specifically, the geometric relationship is determined as follows: define a three-dimensional coordinate vector [X, Y, Z] ^T of any position within the eye box of the display system, and define the coordinates of the target position on the final imaging plane (ie, a two-dimensional coordinate system) is [u,v] ^T , then the pinhole camera model can be obtained:

w[u,v,1] ^T =K[X,Y,Z,1] ^T (3)

Among them, K is the camera intrinsic parameter matrix, which depends on the focal length and pixel displacement of the camera. The camera intrinsic parameter matrix K can be obtained by configuring the camera specifications. Among them, w is the scale factor, which is the scaling ratio between the image plane and the pixel plane, which can be obtained from the actual size of the image and the size of the image captured by the camera.

It should be understood that since the size of the human eye is much smaller than the imaging distance, the pinhole camera model is suitable for the calculation in this embodiment.

It should be understood that, in actual operation, the above-mentioned geometric relationship can be directly determined by using a calibration map for camera calibration: the calibration map can be a solid or virtual undistorted standard pattern. Specifically, the physical calibration map is placed or the virtual calibration map is placed at the imaging distance of the display system. Since the pattern size and ordering of the calibration map are standardized, a camera can be placed at the observation position of the display system, and the calibration map can be photographed to obtain the w and K parameters. If calibration is performed at multiple positions of the eye box, the w and K parameters corresponding to different positions of the eye box can be obtained. Then, the conversion relationship between the different positions of the eye box and the coordinates of the final imaging plane is obtained.

It should be understood that after obtaining the uncorrected left-eye distortion imaging sample and the uncorrected right-eye distortion imaging sample, the target source image sample, the uncorrected left-eye distortion imaging sample and the uncorrected right-eye distortion imaging sample can be obtained according to the Build a distortion model.

It should be understood that the present application does not limit how the distortion model is constructed. Optionally, the correlation between the target source image sample projected by the image generator and the final imaged uncorrected left-eye distortion image sample can be obtained through optical design software or a physical modeling form, and a left-eye distortion model is constructed; obtain the image generator Correlate the projected target source image sample with the final imaged uncorrected right-eye distortion image sample and construct a right-eye distortion model.

Optionally, the target source image samples are compared with the calibration map to the uncorrected distorted image samples to determine the distortion model. The distortion model can also be constructed by comparing the distortion image with the reference image. Specifically, the undistorted calibration map (ie, the target source image sample) is imported into the image generator of the display system, and after optical imaging, the final image placed is a distorted image (ie, an uncorrected distorted image sample). At the same time, a physical or virtual distortion-free calibration map (ie, target source image sample) is placed at the imaging position of the display system, and its shape, size and size need to be proportional to the calibration map displayed by the image generator. Then, place cameras at different positions of the eye box, and shoot the calibration map and the distortion pattern imaged by the display system at the same time, and then convert the captured calibration map and the distortion pattern imaged by the display system in combination with the space conversion relationship mentioned above. After comparing the patterns, the distortion model M of multiple positions of the eye box is obtained. The form of the distortion model may be a parameter table or a mathematical model, and the embodiment of the present application does not limit the form of the distortion model in any way.

The following will briefly describe the system architecture of the embodiments of the present application with reference to Fig. 5 to Fig. 7, so as to better understand the solution of the present application.

FIG. 5 is an example diagram of a system architecture provided by an embodiment of the present application. As shown in FIG. 5 , the system architecture 500 includes: an eye tracker 510 , an image generator 520 , a binocular parallax generator 530 and an optical lens group 540 . It should be understood that the system architecture 500 is only used as an example, and does not constitute a limitation to the present application. Optionally, in the actual system architecture, the above-mentioned eye tracker 510 may not be included, which is not limited in this application. Each of the above components will be described below.

Among them, the eye tracker 510 is used to detect the observer's binocular coordinate information.

The image generator 520 includes an interface for acquiring the target source image from the outside world and a distortion corrector. Among them, the distortion corrector is used to obtain or generate a left-eye distortion model and a right-eye distortion model. Then, the image generator 520 performs pre-distortion processing on the target source image according to the left-eye distortion model and the right-eye distortion model generated by the distortion corrector to generate a left-eye pre-distortion image and a right-eye pre-distortion image. The pixels in the pre-distorted image of the left eye are interleaved with the pixels in the pre-distorted image of the right eye to generate a composite pixel interleaving image. For details, please refer to the description of the display system 200 above. It should be understood that FIG. 5 is only an example, and in an actual system architecture, the above-mentioned distortion corrector may also exist independently of the image generator 520 , which is not limited in this application.

The binocular disparity generator 530, which is installed in the image generator 520, is used to guide the pixels of the left-eye pre-distorted image in the pixel-interleaving synthesis image to be pushed to the left eye of the observer in a spatial division multiplexing manner, and at the same time guide the pixel-interleaving synthesis The pixels of the right eye pre-distorted image in the figure are pushed to the right eye of the observer, so that the undistorted image is simultaneously observed by both eyes. Optionally, the binocular disparity generator 530 may also be independent of the image generator 520, which is not limited in this application. Optionally, the binocular parallax generator 530 may be a parallax barrier, for example, a diaphragm or a lenticular lens; it may also be a directional light source component, which is not limited in this application.

The optical lens group 540, which can also be called an imaging combiner, is mainly used to image the image generated by the image generator 520 and transmit it to the observer's binocular. Preferably, in the present application, the optical mirror group 540 includes a flat mirror and a free-form mirror, and the image generated by the image generator 520 first passes through the flat mirror and then is transmitted to the binoculars through the free-form mirror. However, it should be understood that the design of the optical lens group in the present application is not limited to this.

It should be understood that for different application scenarios, there are also different requirements for the design of the optical lens group 540 .

As an example, FIG. 6 is an example diagram of a system architecture for use in an office entertainment display scenario provided by an embodiment of the present application. As shown in FIG. 6 , the optical mirror group 540 in the system architecture 600 includes a plane mirror 541 and a free-form mirror 542 . In the system architecture 600, the image generator 520 cooperates with the binocular disparity generator 530 to project the pixels in the left-eye pre-distorted image and the pixels in the right-eye pre-distorted image in the pixel interleaving composite image in a spatial division multiplexing manner, and respectively pass The flat mirror 541 and the free-form mirror 542 in the optical mirror group 540 are transmitted to the binocular. It should be understood that the free-form surface mirror 542 has the functions of optical path extension and optical path folding, so that the volume design of the optical lens group 540 is more flexible. And in this example, the free-form mirror 542 is used instead of the concave mirror, because the free-form mirror can be built according to the process requirements, so that the pixels in the left-eye pre-distorted image and the pixels in the right-eye pre-distorted image are accurately pushed. It can reach the eyes of the observer, and has the function of imaging magnification, so that the observer can enjoy a better viewing experience. However, it should be understood that in actual operation, concave mirrors or other optical mirrors may also be used, which is not limited in this application, and the number of optical mirrors is not limited thereto. In addition, the eye tracker 510 in the system architecture 600 enables the dynamic change of the binocular coordinates to be detected and fed back to the image generator 520 when the observer moves significantly, so that the image generator 520 can perform real-time changes according to the coordinate changes. Refresh the left-eye pre-distorted image and the right-eye pre-distorted image. In this way, the observer can always see an undistorted image in the scope of the eye box, which does not damage the viewing experience.

As another example, FIG. 7 is an example diagram of a system architecture for a vehicle head-up display scene provided by an embodiment of the present application. Optionally, the system architecture 700 of the vehicle head-up display scene may be obtained by slightly adjusting the system architecture 600 shown in FIG. 6 . As shown in FIG. 7 , the optical lens group 540 in the system architecture 700 can add an automobile reflective element 543 (such as a windshield) to the optical lens group 540 of the system architecture 600, so that the image can be accurately presented to the driver ( That is, the observer) binocular, to avoid the left and right views do not overlap and interfere with driving. In addition, the eye tracker 510 in the system architecture 700 can ensure that the driver can always watch the undistorted image, especially in scenes with frequent dynamic changes such as vehicle shaking.

In addition, it is easy to see that the solution of the present application only needs one image generator, which has the advantage of a smaller system architecture, which makes it more suitable for application scenarios of vehicle installation or other scenarios with limited volume.

The specific implementation of the binocular distortion correction of the present application will be described below with reference to the above-mentioned system architectures 500 to 700 , taking FIG. 8 as an example.

FIG. 8 is an example diagram of a binocular distortion correction method provided by an embodiment of the present application. As shown in FIG. 8 , the method 800 includes steps S810 to S850. It should be understood that the embodiments of the present application do not limit the sequence of the above steps, and any solution that can be implemented in the present application through any sequence of the above steps falls within the protection scope of the present application. It should also be understood that FIG. 8 is only an optional example, and does not constitute a limitation to the present application. These steps are described in detail below.

S810, the eye tracker detects the observer's binocular coordinate information.

S820, the distortion corrector acquires a distortion model.

Optionally, the distortion corrector can directly read the left-eye distortion model and the right-eye distortion model corresponding to the current left-eye and right-eye coordinates from the parameter table in the system according to the current binocular coordinate information, as described above.

Optionally, the distortion model obtained by the distortion corrector may also be a distortion model generated by itself, as described above.

It should be understood that, in the embodiment, the distortion corrector may exist in the image generator, as shown in FIG. 5 to FIG. 7 , or may exist independently of the image generator, which is not limited in this application.

S830, the image generator generates a pre-distorted image for the left eye and a pre-distorted image for the right eye.

Specifically, the image generator needs to generate a left-eye pre-distortion image and a right-eye pre-distortion image according to the target source image, the left-eye distortion model, and the right-eye distortion model. It should be understood that the target source image is the target source image that the display system needs to display during actual use, that is, the image that needs to be displayed for the user.

In an implementation manner, the target source image is a source image captured at a target viewing angle, and the present application does not limit the target viewing angle. The left-eye pre-distorted image can be generated according to the target source image and the left-eye distortion model, and the right-eye pre-distorted image can be generated according to the target source image and the right-eye distortion model (achieved by formula (2)).

In this case, pre-distortion processing is performed according to the same source image (ie, target source image) to obtain a left-eye pre-distortion image and a right-eye pre-distortion image, so that the image observed at the binocular is a 2D image without distortion.

In another implementation manner, the target source image includes a left-eye view target source image and a right-eye view target source image, and the acquisition methods of the left-eye view target source image and the right-eye view target source image are not limited, and refer to the above description. The left eye pre-distorted image can be generated according to the left eye perspective target source image and the left eye distortion model, and the right eye pre-distorted image can be generated according to the right eye perspective target source image and the right eye distortion model (achieved by formula (2)).

In this case, the images from different perspectives (ie, the target source image from the left eye perspective and the target source image from the right eye perspective) are respectively pre-distorted to obtain the left eye pre-distorted image and the right eye pre-distorted image, so that the image observed in the binocular can be obtained. For distortion-free 3D imaging.

S840, the image generator generates a pixel-interleaved composite image.

For details, reference may be made to the description of FIG. 3 above, which will not be repeated here.

S850, the pixel interleaving composite image is projected by space division multiplexing, and pushed to the observer's binocular through the optical mirror group.

It should be understood that the space division multiplexing method is implemented with the cooperation of the binocular disparity generator, and reference may be made to the description in the part of FIG. 4 above.

Specifically, with the cooperation of the binocular parallax generator, the pixels of the left eye pre-distorted image in the pixel interleaving composite image generated by the image generator are pushed to the left eye of the observer through the optical lens group, and the pixels in the pixel interleaving composite image are pushed to the left eye of the observer. The pixels of the pre-distorted image of the right eye are pushed to the right eye of the observer through the optical lens group, so that the observer's binocular can view the image without distortion.

In the embodiment of the present application, a pixel-interleaved composite image is generated by interleaving the pixels in the left-eye pre-distorted image with the pixels in the right-eye pre-distorted image, and then, under the action of the binocular disparity generator, the pixel-interleaved composite image is separated by space. The multiplexing method is pushed to the binocular, which can improve the user's visual experience while reducing the computational pressure of the data reading system.

FIG. 9 is an example diagram of a display method for binocular distortion correction provided by an embodiment of the present application. As shown in FIG. 9, the method 900 includes steps S910 to S930, which will be described in detail below.

S910, a left eye pre-distorted image and a right eye pre-distorted image are generated.

Optionally, before generating the left-eye pre-distortion image and the right-eye pre-distortion image, the method 900 may further include: acquiring a left-eye distortion model and a right-eye distortion model. Among them, the left eye distortion model is generated from the target source image sample and the uncorrected left eye distortion image sample, and the right eye distortion model is generated from the target source image sample and the uncorrected right eye distortion image sample.

Optionally, generating the pre-distorted image for the left eye and the pre-distorted image for the right eye includes: acquiring a target source image; and generating the pre-distorted image for the left eye and the pre-distorted image for the right eye according to the target source image, the left eye distortion model and the right eye distortion model.

Optionally, the target source image is a source image captured at a target viewing angle. Then, generating the left-eye pre-distortion image and the right-eye pre-distortion image according to the target source image, the left-eye distortion model and the right-eye distortion model includes: generating the left-eye pre-distortion image according to the target source image and the left-eye distortion model, and generating the right-eye pre-distortion image according to the target source image and the right-eye distortion model. Distorted image.

Optionally, the target source image includes a left eye perspective target source image and a right eye perspective target source image. Then, generating the left-eye pre-distortion image and the right-eye pre-distortion image according to the target source image, the left-eye distortion model and the right-eye distortion model includes: generating the left-eye pre-distortion image according to the left-eye perspective target source image and the left-eye distortion model; The model generates a right eye predistorted image.

Optionally, before acquiring the left-eye distortion model and the right-eye distortion model, the method 900 may further include: detecting the binocular coordinates of the observer. Then, obtaining the left-eye distortion model and the right-eye distortion model includes: obtaining the left-eye distortion model and the right-eye distortion model according to the observer's binocular coordinates.

Optionally, the uncorrected left-eye distortion imaging sample and the uncorrected right-eye distortion imaging sample are obtained according to the spatial transformation relationship between the binocular coordinates of the observer and the coordinates of the target position on the final imaging plane, and the target position It is the intersection point of the extension line of the left eye line of sight and the right eye line of sight.

S920: Interleave the pixels in the pre-distorted image for the left eye with the pixels in the pre-distorted image for the right eye to generate a composite image with interleaving of pixels.

Optionally, interleaving the pixels in the left-eye pre-distorted image with the pixels in the right-eye pre-distorted image to generate a pixel interleaving composite image includes: overlapping the left-eye pre-distorted image and the right-eye pre-distorted image; determining an envelope range, and the envelope range covers The effective pixels in the left-eye pre-distorted image and the right-eye pre-distorted image; the pixels in the left-eye pre-distorted image and the pixels in the right-eye pre-distorted image are interleaved within the envelope range to generate a pixel interleaving composite image.

S930, under the action of the binocular parallax generator, push the pixel interlaced composite image to the observer's binocular through the optical lens group.

Among them, the pixels of the left eye pre-distorted image in the pixel interleaving composite image are pushed to the left eye of the observer, and the pixels of the right eye pre-distorted image in the pixel interleaving composite image are pushed to the observer's right eye.

Optionally, the optical mirror group may include a flat mirror and a free-form mirror.

Optionally, the binocular parallax generator may be a parallax barrier or a directional light source component.

Alternatively, the parallax barrier may be a diaphragm or a lenticular lens.

FIG. 10 is an exemplary block diagram of a hardware structure of an apparatus provided by an embodiment of the present application. The apparatus 1000 (the apparatus 1000 may specifically be a computer device) includes a memory 1010 , a processor 1020 , a communication interface 1030 and a bus 1040 . The memory 1010 , the processor 1020 and the communication interface 1030 are connected to each other through the bus 1040 for communication.

The memory 1010 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1010 may store a program, and when the program stored in the memory 1010 is executed by the processor 1020, the processor 1020 is configured to execute each step of the display method of the embodiment of the present application.

The processor 1020 may adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processor (graphics processing unit, GPU), or one or more The integrated circuit is used to execute the relevant program to realize the display method of the method embodiment of the present application.

The processor 1020 may also be an integrated circuit chip with signal processing capability. In the implementation process, the display method of the present application may be implemented by an integrated logic circuit of hardware in the processor 1020 or an instruction in the form of software.

The above-mentioned processor 1020 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, Discrete gate or transistor logic devices, discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 1010, and the processor 1020 reads the information in the memory 1010, and combines its hardware to complete the functions required to be performed by the modules included in the apparatus of the embodiments of the present application, or to execute the display methods of the method embodiments of the present application.

The communication interface 1030 implements communication between the apparatus 1100 and other devices or a communication network using a transceiving device such as, but not limited to, a transceiver.

Bus 1040 may include a pathway for communicating information between various components of device 1000 (eg, memory 1010, processor 1020, communication interface 1030).

An embodiment of the present application further provides a controller, including an input and output interface, a processor and a memory, where the processor is used for controlling the input and output interface to send and receive signals or information, the memory is used for storing a computer program, and the processor is used for The computer program is called and executed from the memory, so that the controller executes the display method for binocular distortion correction according to the method embodiment of the present application.

The embodiments of the present application also provide a vehicle-mounted system, including the display system for binocular distortion correction according to the device embodiments of the present application.

The embodiment of the present application also provides a desktop display system, including the display system for binocular distortion correction according to the device embodiment of the present application.

The embodiments of the present application also provide a vehicle, including the display system for binocular distortion correction according to the device embodiments of the present application. Optionally, the vehicle involved in this application may be a traditional internal combustion engine vehicle, a hybrid electric vehicle, a pure electric vehicle, a centralized drive vehicle, a distributed drive vehicle, etc., which is not limited in this application.

The embodiments of the present application also provide a computer program product containing instructions, when the computer program product runs on a computer, the computer program product enables the computer to execute the display method for binocular distortion correction of the above method embodiments.

Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program, and the computer program includes a display for binocular distortion correction for executing the method embodiments of the present application method instruction.

The embodiments of the present application also provide a computer program, where the computer program includes instructions for executing the display method for binocular distortion correction according to the method embodiments of the present application.

An embodiment of the present application further provides a chip, where the chip includes a processor and a data interface, the processor reads an instruction stored in a memory through the data interface, and executes the method for binocular distortion of the method embodiment of the present application. Corrected display method.

Optionally, as an implementation manner, the chip may further include a memory, in which instructions are stored, the processor is configured to execute the instructions stored in the memory, and when the instructions are executed, the The processor is configured to execute the display method for binocular distortion correction according to the method embodiment of the present application.

Embodiments of the present application further provide a chip system, where the chip system includes at least one processor for supporting functions involved in implementing certain implementations of the display method for binocular distortion correction in the above method embodiments, for example , such as receiving or processing data and/or information involved in the above methods.

In a possible design, the chip system further includes a memory for storing program instructions and data, and the memory is located inside the processor or outside the processor. The chip system may be composed of chips, or may include chips and other discrete devices.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (26)

1. A display system for binocular distortion correction, comprising: an image generator, a binocular parallax generator and an optical lens group;

   The image generator is configured to generate a left-eye pre-distorted image and a right-eye pre-distorted image, and interleave the pixels in the left-eye pre-distorted image with the pixels in the right-eye pre-distorted image to generate a pixel-interleaved composite image; The eye parallax generator causes the pixel interleaving composite image to be pushed to the observer's binocular through the optical lens group, wherein the pixels of the left eye pre-distorted image in the pixel interleaving composite image are pushed to the observer's binocular. For the left eye, the pixels of the right eye predistorted image in the pixel-interleaved composite image are pushed to the observer's right eye.
2. The display system of claim 1, wherein the image generator is further configured to:

   overlapping the left eye pre-distorted image and the right eye pre-distorted image;

   determining an envelope range, the envelope range covers valid pixels in the left-eye pre-distorted image and the right-eye pre-distorted image;

   The pixels in the left-eye pre-distorted image and the pixels in the right-eye pre-distorted image are interleaved within the envelope range to generate a pixel interleaving composite image.
3. The display system according to claim 1 or 2, wherein the display system further comprises: a distortion corrector; the distortion corrector is used for:

   Obtain a left-eye distortion model and a right-eye distortion model, the left-eye distortion model is generated from the target source image sample and the uncorrected left-eye distortion image sample, and the right-eye distortion model is based on the target source image sample and the uncorrected right-eye distortion generated from the imaging sample;

   The image generator is also used to,

   Acquiring a target source image; generating the left-eye pre-distortion image and the right-eye pre-distortion image according to the target source image, the left-eye distortion model and the right-eye distortion model.
4. The display system according to claim 3, wherein the target source image is a source image captured at a target viewing angle; the image generator is further configured to:

   The left-eye pre-distortion image is generated according to the target source image and the left-eye distortion model, and the right-eye pre-distortion image is generated according to the target source image and the right-eye distortion model.
5. The display system according to claim 3, wherein the target source image comprises a left eye perspective target source image and a right eye perspective target source image;

   The image generator is also used to,

   The left eye pre-distorted image is generated according to the left eye perspective target source image and the left eye distortion model, and the right eye pre-distorted image is generated according to the right eye perspective target source image and the right eye distortion model.
6. The display system according to claim 4 or 5, wherein the display system further comprises: an eye tracker; the eye tracker is used for:

   detecting the binocular coordinates of the observer;

   The distortion corrector is further configured to acquire the left-eye distortion model and the right-eye distortion model according to the observer's binocular coordinates.
7. The display system according to claim 6, wherein the uncorrected left-eye distortion imaging sample and the uncorrected right-eye distortion imaging sample are based on the binocular coordinates of the observer and the final imaging plane. The target position is obtained from the spatial transformation relationship between the coordinates of the target position of , where the target position is the intersection of the extension line of the left eye line of sight and the right eye line of sight.
8. The display system according to any one of claims 1 to 7, wherein the optical mirror group comprises a flat mirror and a free-form mirror.
9. The display system according to any one of claims 1 to 8, wherein the binocular parallax generator is a parallax barrier or a directional light source component.
10. The display system according to claim 9, wherein the parallax barrier is a diaphragm or a lenticular lens.
11. The display system according to any one of claims 1 to 10, wherein the display system is an office entertainment display system or a vehicle head-up display system.
12. A display method for binocular distortion correction, comprising:

    Generate a left eye pre-distorted image and a right eye pre-distorted image;

    Interleaving the pixels in the left-eye pre-distorted image with the pixels in the right-eye pre-distorted image to generate a composite image of pixel interleaving;

    Pushing the pixel-interleaved composite image to the observer's binocular, wherein the pixels of the left-eye pre-distorted image in the pixel-interleaved composite image are pushed to the observer's left eye, and the pixel-interleaved composite image of the right-eye image is pushed to the observer's left eye. The pixels of the predistorted image are pushed to the right eye of the viewer.
13. The display method according to claim 12, wherein the interleaving of the pixels in the left-eye pre-distorted image and the pixels in the right-eye pre-distorted image to generate a pixel interleaving composite image comprises:

    overlapping the left eye pre-distorted image and the right eye pre-distorted image;

    determining an envelope range, the envelope range covers valid pixels in the left-eye pre-distorted image and the right-eye pre-distorted image;

    The pixels in the left-eye pre-distorted image and the pixels in the right-eye pre-distorted image are interleaved within the envelope range to generate a pixel interleaving composite image.
14. The display method according to claim 12 or 13, wherein the method further comprises:

    Obtain a left-eye distortion model and a right-eye distortion model, the left-eye distortion model is generated from the target source image sample and the uncorrected left-eye distortion image sample, and the right-eye distortion model is based on the target source image sample and the uncorrected right-eye distortion generated from the imaging sample;

    The generating of the left-eye pre-distorted image and the right-eye pre-distorted image includes:

    Get the target source image;

    The left-eye pre-distortion image and the right-eye pre-distortion image are generated according to the target source image, the left-eye distortion model and the right-eye distortion model.
15. The display method according to claim 14, wherein the target source image is a source image captured at a target viewing angle; the generating the target source image according to the target source image, the left-eye distortion model and the right-eye distortion model The left-eye pre-distorted image and the right-eye pre-distorted image include:

    The left-eye pre-distortion image is generated according to the target source image and the left-eye distortion model, and the right-eye pre-distortion image is generated according to the target source image and the right-eye distortion model.
16. display method according to claim 14, is characterized in that, described target source image comprises left eye angle target source image and right eye angle target source image;

    The generating the left-eye pre-distortion image and the right-eye pre-distortion image according to the target source image, the left-eye distortion model and the right-eye distortion model includes:

    The left eye pre-distorted image is generated according to the left eye perspective target source image and the left eye distortion model, and the right eye pre-distorted image is generated according to the right eye perspective target source image and the right eye distortion model.
17. The display method according to claim 15 or 16, wherein the method further comprises:

    detecting the binocular coordinates of the observer;

    The obtaining of the left-eye distortion model and the right-eye distortion model includes:

    The left-eye distortion model and the right-eye distortion model are acquired according to the observer's binocular coordinates.
18. The display method according to claim 17, wherein the uncorrected left-eye distortion imaging sample and the uncorrected right-eye distortion imaging sample are based on the binocular coordinates of the observer and the final imaging plane. The target position is obtained from the spatial transformation relationship between the coordinates of the target position of , where the target position is the intersection of the extension line of the left eye line of sight and the right eye line of sight.
19. The display method according to any one of claims 12 to 18, wherein the method further comprises:

    The binocular parallax generator is used to push the pixel interlaced composite image to the observer's binocular, wherein the binocular parallax generator is a parallax barrier or a directional light source component.
20. The display method according to claim 19, wherein the parallax barrier is a diaphragm or a lenticular lens.
21. A controller is characterized in that it comprises an input and output interface, a processor and a memory, the processor is used to control the input and output interface to send and receive signals or information, the memory is used to store a computer program, and the processor is used to retrieve a computer program from the memory The computer program is called and executed, so that the controller executes the display method for binocular distortion correction according to any one of claims 12 to 18.
22. An in-vehicle system, characterized by comprising the display system for binocular distortion correction according to any one of claims 1 to 11.
23. A desktop display system, characterized in that it includes the display system for binocular distortion correction according to any one of claims 1 to 11.
24. A vehicle, characterized by comprising the display system for binocular distortion correction according to any one of claims 1 to 11.
25. A computer program, characterized in that the computer program includes instructions for executing the display method for binocular distortion correction according to any one of claims 12 to 18.
26. A computer-readable medium, characterized by being used for storing a computer program, wherein the computer program includes instructions for executing the display method for binocular distortion correction according to any one of claims 12 to 18.

Images