WO2021197189A1

WO2021197189A1 - Augmented reality-based information display method, system and apparatus, and projection device

Info

Publication number: WO2021197189A1
Application number: PCT/CN2021/082943
Authority: WO
Inventors: 余新; 康瑞; 邓岳慈; 弓殷强; 赵鹏
Original assignee: 深圳光峰科技股份有限公司
Priority date: 2020-03-31
Filing date: 2021-03-25
Publication date: 2021-10-07
Also published as: CN113467600A

Abstract

An augmented reality-based information display method, system and apparatus, a projection device and a storage medium. The method comprises: obtaining real scene information collected by a scene sensing device (S110); obtaining a target display area determined on the basis of a first spatial pose of a user, the first spatial pose being determined on the basis of an eye tracking device (S120); obtaining coordinate transformation rules corresponding to real scene information mapped to the target display area (S130); generating driving guidance information on the basis of the real scene information (S140); and displaying the driving guidance information in position corresponding to the target display area on the basis of the coordinate transformation rules (S150). Driving guidance information is displayed, by means of the coordinate transformation rules, in the position corresponding to the target display area determined on the basis of the first spatial pose of the user as determined by the eye tracking device. The first spatial pose of the user is tracked in real-time by the eye tracking device, which enables adaptation to changes to the eye pose, and the display area and display position of the driving guidance information are dynamically adjusted.

Description

Information display method, system, device and projection equipment based on augmented reality

Technical field

This application relates to the technical field of coordinate transformation, and more specifically, to an information display method, system, device, projection device, and storage medium based on augmented reality.

Background technique

HUD (head up display) is a head-up display (or head-up display), which can display important information on a piece of transparent glass in front of the line of sight. Look down at the data in the dashboard, so as to avoid that the pilot cannot observe the environmental information in the field ahead of the flight when viewing the data in the dashboard. In order to reduce accidents caused by users looking down at the dashboard or center console, HUD was introduced from the aircraft to the automotive field.

However, the existing HUD display method lacks intelligence. Take car driving as an example. With the addition of more driving assistance information such as road conditions, navigation, and danger warning, the user's body and head will change or change with the road conditions. It is a personal activity that causes the user's spatial posture to change. In this case, it may cause the user to see that the image displayed by the HUD is offset from the real scene image.

Summary of the invention

In view of the above-mentioned problems, this application proposes an augmented reality-based information display method, system, device, projection equipment, and storage medium to improve the above-mentioned problems.

In the first aspect, an embodiment of the present application provides an information display method based on augmented reality. The method includes: acquiring real scene information collected by a scene sensing device; acquiring a target display area determined based on a user's first spatial pose, so The first spatial pose is determined based on the human eye tracking device; acquiring the coordinate transformation rule corresponding to the real scene information mapped to the target display area; generating driving guidance information based on the real scene information; The driving guide information is displayed at the corresponding position of the target display area.

In a second aspect, an embodiment of the present application provides an augmented reality information display device. The information display device includes an image perception module, a coordinate transformation module, and a display module: the image perception module is used to obtain the real scene collected by the image perception device. Information; the coordinate transformation module is used to obtain the target display area determined based on the user's first spatial pose, the first spatial pose is determined based on the eye tracking device; the coordinate transformation module is also used to obtain the The real scene information is mapped to the coordinate transformation rule corresponding to the target display area; the display module is used to generate driving guidance information based on the real scene information; the display module is also used to transform the driving guidance information based on the coordinate transformation rule Displayed in the corresponding position of the target display area.

In a third aspect, an embodiment of the present application provides an augmented reality vehicle-mounted information display system. The system includes: a scene sensing device for collecting real-world information of the vehicle's external environment; and a human eye tracking device for acquiring the user's line of sight Range of movement; an image processing device for acquiring a first spatial pose of the user based on the range of movement of the line of sight, acquiring a target display area determined based on the first spatial pose, and acquiring the real scene information mapped to the The coordinate transformation rule corresponding to the target display area is to generate driving guidance information based on the real scene information, and the target position coordinates of the driving guidance information displayed in the target display area are generated based on the coordinate transformation rule; the HUD display device is used to display The driving guidance information is displayed at the target position coordinates of the target display area.

In a fourth aspect, an embodiment of the present application provides a projection device, including one or more processors and a memory; one or more programs are stored in the memory and configured to be processed by the one or more The one or more programs are configured to execute the method described in the first aspect.

In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having program code stored in the computer-readable storage medium, wherein the method described in the first aspect is executed when the program code is running.

The present application provides an information display method, system, device, projection device, and storage medium based on augmented reality, which acquires real scene information collected by an image sensing device, and then acquires a target display area determined based on the user's first spatial pose, The first spatial pose is determined based on the human eye tracking device, and then the real scene information is mapped to the coordinate transformation rules corresponding to the target display area, and the driving guidance information is generated based on the real scene information, and then the driving guidance information is displayed on the target display based on the coordinate transformation rules The corresponding location of the area. As a result, the driving guidance information generated based on the real scene information is displayed in the corresponding position of the target display area determined based on the user's first spatial pose determined by the eye tracking device through the coordinate transformation rule through the above method. The eye tracking device tracks the user's first spatial pose in real time, so that it can adapt to changes in the human eye pose, dynamically adjust the display area and display position of the driving guidance information, and realize that the user can accurately and conveniently drive Viewing the virtual driving guidance information corresponding to the driving scene improves the safety and comfort of driving, thereby enhancing the user experience.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

Fig. 1 shows a method flowchart of an augmented reality-based information display method proposed by an embodiment of the present application.

Fig. 2 shows an example of the structure of an augmented reality-based vehicle information display system based on the augmented reality-based information display method provided by this embodiment.

FIG. 3 shows an example diagram of the HUD virtual image plane of the HUD display device in this embodiment.

Figure 4 shows a schematic diagram of the relationship between the driver's eyes and the HUD virtual image plane in this embodiment.

FIG. 5 shows an example diagram of displaying driving guidance information through the augmented reality-based information display system proposed in this application in a dangerous scenario provided by this embodiment.

Fig. 6 shows a method flowchart of an augmented reality-based information display method proposed by another embodiment of the present application.

FIG. 7 shows an example diagram of the display effect of the information display system based on augmented reality provided by this embodiment.

FIG. 8 shows another example diagram of the display effect of the information display system based on augmented reality provided by this embodiment.

FIG. 9 shows a method flowchart of an augmented reality-based information display method proposed by another embodiment of the present application.

FIG. 10 shows an example diagram of adjusting the target display area based on the change vector provided by this embodiment.

FIG. 11 shows an example diagram of the processing procedure of the information display method based on enhanced display proposed in this embodiment.

FIG. 12 shows a structural block diagram of an information display device based on augmented reality proposed by an embodiment of the present application.

Fig. 13 shows a structural block diagram of a projection device of the present application for executing an augmented reality-based information display method according to an embodiment of the present application.

Fig. 14 shows a storage unit for storing or carrying program code for implementing an augmented reality-based information display method according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

HUD is mainly divided into two types: rear-mounted (also known as Combine HUD, C-type HUD) and front-mounted (also known as Windshield HUD, W-type HUD). Among them, the front-mounted HUD uses the windshield as a combiner to project the content required by the driver to the front windshield through the optical system. Driving safety and driving comfort. However, some existing HUD devices only display virtual information in front of the driver's line of sight, and are not integrated with the real environment. With the addition of more driving assistance information such as road conditions, navigation, and hazard warnings, the mismatch between this virtual content and the real scene will cause the driver's attention to be distracted.

Augmented Reality (AR) is a technology that ingeniously integrates virtual information with the real world.

As a way, with the development of autonomous driving, augmented reality, and mixed reality technology, AR technology can be introduced into the HUD field. AR-HUD can solve the separation of traditional HUD virtual information and actual scenes through the combination of AR technology and front-mounted HUD. The problem of mismatch, while enriching the HUD display content, improves the safety and comfort of driving. However, the existing HUD display method lacks intelligence. Take car driving as an example. With the addition of more driving assistance information such as road conditions, navigation, and danger warning, the user's body and head will change or change with the road conditions. It is a personal activity that causes the user's spatial posture to change. In this case, it may cause the user to see that the image displayed by the HUD is offset from the real scene image.

Therefore, in order to improve the above problems, the inventor proposes the real scene information that can be collected by the image sensing device provided in this application, and then obtains the target display area determined based on the user's first spatial pose, where the first spatial pose is Based on the determination of the human eye tracking device, the real scene information is mapped to the coordinate transformation rule corresponding to the target display area, and then the driving guidance information is generated based on the real scene information, and then the driving guidance information is displayed in the corresponding position of the target display area based on the coordinate transformation rule. The driving guidance information generated based on real scene information is displayed in the corresponding position of the target display area determined based on the user's first spatial pose determined by the eye tracking device through the coordinate transformation rule, and the real-time tracking device is realized by the eye tracking device. Tracking the user’s first spatial posture makes it possible to adapt to changes in the human eye’s posture and dynamically adjust the display area and display position of the driving guidance information, so that the user can accurately and conveniently view the corresponding driving scene during driving. The virtual driving guidance information improves the safety and comfort of driving, thereby enhancing the user experience.

Hereinafter, each embodiment of the present application will be described in detail with reference to the accompanying drawings.

Please refer to FIG. 1, which is a method flowchart of an augmented reality-based information display method provided by an embodiment of this application. The method of this embodiment may be executed by an augmented reality-based device for processing real-scene information, and the device may be implemented by hardware and/or software, and the method includes:

Step S110: Acquire real scene information collected by the scene sensing device.

Among them, the real scene information in the embodiment of the present application may be real scene information corresponding to multiple scenes. Optionally, multiple scenes may include, but are not limited to, driving scenes, travel scenes, and outdoor activity scenes. For example, if it is a driving scene, the real scene information can include lanes, signs, dangerous pedestrians (such as vulnerable groups such as blind people, elderly people walking alone, pregnant women, or children), vehicles, etc.; if it is a tourist scene, the real scene information can include tourist destination signs. , Tourist routes, tourist attractions information and tourist attractions weather information, etc.; if it is an outdoor activity scene, the real scene information can include current location information and nearby convenience store information.

Optionally, the scene sensing device may include sensing devices such as lasers and infrared radars, and may also include image acquisition devices such as cameras (including monocular cameras, binocular cameras, RGB-D cameras, etc.). As a way, the real scene information corresponding to the current scene can be acquired through the scene sensing device. For example, suppose the current scene is a driving scene, and the scene sensing device is a camera. The camera can be installed on the car (optionally, the installation position can be adjusted according to the style and structure of the car or the actual needs), so that the camera can obtain the Real-life information related to driving. Among them, for the collection principle and implementation of the scene sensing device (including laser, infrared radar, or camera) collecting real scene information, please refer to related technologies, which will not be repeated here.

Step S120: Obtain a target display area determined based on a first spatial pose of the user, where the first spatial pose is determined based on the eye tracking device.

In this embodiment, the first spatial pose may be a human eye pose determined based on the human eye tracking device. It is understandable that if the user's line of sight range changes, the corresponding human eye pose can also change, and if the user's body rotates and the eye line of sight does not change, the human eye pose can remain unchanged In this way, because the user’s eyes can see the real-life information that needs attention in the driving scene (for example, road conditions, dangerous pedestrians and vehicles, etc.), safe driving is still possible, so as a way, the first space The pose may be determined based on the user's current eye pose. In this way, the target display area determined based on the user's current eye pose may be obtained. Optionally, the eye tracking device may be a device with a camera function, such as a camera, and the details may not be limited.

Optionally, if the user's eye pose changes, but the range of the change is small (for example, less than a preset threshold), the user's first spatial pose may be the sitting posture of the user in a driving state, Or it is the sitting posture after adjusting the seat (here can be the current user adjusting the seat for the first time). It is understandable that the user's sitting posture is different, and the corresponding spatial posture is different. As a way, the sitting posture of the user after adjusting the seat can be used as the user's first spatial posture. Optionally, if the change range of the user's eye pose is not less than a preset threshold, the user's first spatial pose may be re-determined according to the user's current eye pose.

In this embodiment, the target display area is an area for displaying virtual image information corresponding to real scene information. Taking a driving scene as an example, the target display area may be an area on the windshield of a car for displaying projected virtual image information corresponding to real scene information. Optionally, the target display areas corresponding to different spatial poses of the same user may be different, and the target display areas corresponding to the spatial poses of different users may be different.

In order to eliminate the display difference between the position where the virtual image information corresponding to the real scene information is displayed and the actual position of the real scene information, as a way, the target display area determined based on the user's first spatial pose can be acquired, so that the target display area can be displayed in the target display area. Displaying the virtual image information corresponding to the real scene information reduces the foregoing display difference, thereby improving the accuracy of the display position of the virtual image information corresponding to the real scene information.

Step S130: Obtain a coordinate transformation rule corresponding to the mapping of the real scene information to the target display area.

Among them, the coordinate transformation rule can be used to map the coordinates of the real scene information to the corresponding coordinates of the target display area. As a way, when the real scene information and the target display area are acquired, the coordinate transformation rules corresponding to the real scene information mapped to the target display area can be acquired, so that the subsequent driving guidance information corresponding to the real scene information can be accurately determined based on the coordinate transformation rules. Is displayed in the corresponding position of the target display area.

Optionally, the coordinate transformation rule may include a first transformation matrix and a second transformation matrix. Wherein, the first transformation matrix can be used to determine the reference world coordinates corresponding to the coordinates of the real scene information collected by the scene sensing device, and the second transformation matrix can be used to convert the reference world coordinates into an offset from the user's perspective in the target display area Matching view coordinates. Wherein, the reference world coordinates can be understood as the relative position coordinates of the real scene information in the established coordinate system corresponding to the scene sensing device. Optionally, the reference world coordinates in this embodiment can be understood as the world coordinates that are relatively stationary with the car. View coordinates can be understood as the relative position coordinates of the reference world coordinates in the coordinate system corresponding to the target display area.

As an implementation manner, the first transformation matrix and the second transformation matrix can be obtained, and then the product of the parameter represented by the first transformation matrix and the parameter represented by the second transformation matrix is mapped to the coordinates corresponding to the target display area as real scene information Transformation rules.

Optionally, the first transformation matrix may include a first rotation matrix and a first translation vector. The first rotation matrix may be used to rotate the coordinates of the real scene information collected by the scene sensing device, and the first translation vector may be used to translate the coordinates. As a way, the reference world coordinates corresponding to the coordinates of the real scene information collected by the scene sensing device may be determined based on the first rotation matrix and the first translation vector.

Optionally, the second transformation matrix may include a viewing angle offset matrix, a transpose matrix, and a projection matrix. Among them, the projection matrix can be used to determine the mapping range of the real scene information to the target display area, the viewing angle offset matrix can be used to determine the degree of deviation of the user's viewing angle detected by the eye tracking device, and the transposition matrix can be used to determine The relative position of the driving guidance information is displayed within the mapping range. As a way, the reference world coordinates may be converted into view coordinates matching the offset of the user's viewing angle in the target display area based on the viewing angle offset matrix, the transposed matrix, and the projection matrix. Wherein, the viewing angle offset matrix may include the first view coordinates, the transposition matrix represents the transposition facing the first transformation matrix, the transposition matrix may include a spatial unit vector located in the target display area, and the projection matrix may include field angle parameters, which may be Optionally, the field of view parameter may include a distance parameter and a scale parameter associated with the user's viewing angle.

The following takes a driving scene as an example to illustrate this embodiment as an example:

Please refer to FIG. 2, which is a structural example diagram of a vehicle-mounted information display system based on augmented reality that is applicable to the method for displaying information based on augmented reality provided by this embodiment. As shown in FIG. 2, the vehicle-mounted information display system based on augmented reality may include a scene perception device, an image processing device, a HUD display device, and a human eye tracking device. Among them, the scene sensing device can be used to collect real scene information of the external environment of the vehicle. The eye tracking device can be used to obtain the user's line of sight movement range. The image processing device can be used to obtain the user's first spatial pose based on the movement range of the line of sight, obtain the target display area determined based on the first spatial pose, obtain the coordinate transformation rule corresponding to the real scene information mapped to the target display area, and generate based on the real scene information The driving guidance information is generated based on the coordinate transformation rules to display the target position coordinates of the driving guidance information in the target display area. The HUD display device can be used to display driving guidance information to the target position coordinates of the target display area.

The image processing device may be the processor chip of the vehicle system, or the processing chip of an independent vehicle computer system, or the processor chip integrated in the scene sensing device (such as lidar), etc., which is not limited herein.

In an implementation manner, the vehicle-mounted information display system may include a car, a driver, a human eye tracking device, a scene perception device, an image processing device, and a HUD display device with AR-HUD function. As an implementation manner, the scene sensing device can be installed on the car and can obtain driving-related scene information (also can be understood as the aforementioned real scene information), the driver sits in the driving position of the car, and the eye tracking device is installed in the car. In the car, the eye tracking device can track the reasonable position of the basic movement range of the driver's eyes. Optionally, the reasonable position here can be understood as the position where the driver's line of sight matches the driving demand during driving, such as to the left Turning, turning right or turning backward, etc., the specific direction and angle of turning may not be limited. The HUD display device is installed on the front windshield of the car, and the position of the HUD display device can be adjusted so that the driver's eyes can see the entire virtual image corresponding to the driving scene information. Optionally, the image processing device can adapt to the change of the driver’s eye spatial pose. In this way, the image processing device can convert the real-world information collected by the scene-sensing device into real-time information that is fused with the real-world information of the real scene in real time. The image is sent to the HUD display device for display.

As a way, after the scene sensing device collects the real scene information, it can obtain the position coordinates of the real scene information in the world coordinate system (O-xyz as shown in Figure 2) based on GPS positioning and other location acquisition methods, and then can be based on the car Select the world coordinate origin and coordinate axis direction for the traveling direction, and determine the reference world coordinate system relative to the car according to the world coordinate origin and the coordinate axis direction. By determining the reference world coordinate system relative to the car, the reference world coordinate system can be obtained The following reference world coordinates corresponding to the coordinates of the real scene information. Among them, the method of selecting the origin of the world coordinate and the direction of the coordinate axis can refer to the related technology, which will not be repeated here. It should be noted that the reference world coordinate system can be understood as a coordinate system obtained after rotating and/or translating the world coordinate system.

For example, as an implementation manner, in the case that a reference world coordinate system that is relatively stationary with the car is determined, the spatial pose of the scene sensing device in the reference world coordinate system can be obtained. In this way, the scene can be sensed The device calculates the perception module transformation matrix M (that is, the aforementioned first transformation matrix) based on the spatial pose in the reference world coordinate system. Exemplarily, it can be assumed that the process of changing the world coordinate system to the reference world coordinate system includes the first rotation matrix (also can be understood as the total rotation matrix of the scene sensing device) R _M and the first translation vector T _M , optionally , sensing module transformation between the matrix M and the first rotation matrix R _M T _M and the first translation vector may be expressed as:

in,

Among them, R _Mx , R _My , and R _Mz are the rotation matrices of the perception module transformation matrix M around the x-axis, y-axis, and z-axis of the world coordinate system, respectively. The Euler angles of rotation are α _M , β _M , γ _M , (T _Mx , T _My , T _Mz ) are the coordinates of the real scene information in the reference world coordinate system.

Optionally, in the reference world coordinate system, the spatial pose of the virtual image displayed on the plane where the HUD display device is located and the spatial pose of the driver’s eyes can be determined. In this way, it can be based on the spatial pose of the displayed virtual image and driving The spatial pose of the eyes of the driver (that is, the pose of the eyes of the driver) obtains the aforementioned second transformation matrix.

The second transformation matrix (here also to be understood as a virtual image of the perspective matrix) C may include a viewing angle offset matrix T, N ^T a transposed matrix and a projection matrix P, viewing angle offset matrix T may be determined by the human eye pose of the driver , transposed matrix N ^T may be displayed by means of virtual HUD image plane to determine the spatial position and orientation, the projection matrix P can be determined as a virtual device common plane spatial position and orientation of the display by the human eye pose HUD and a driver. As a way of viewing angle offset matrix T, the relationship between N ^T a transposed matrix and the projection matrix P can be expressed as three:

C=PN ^T T.

The viewing angle offset matrix T may include the first view coordinates, and the first view coordinates are the position coordinates of the driver's eye pose in the reference world coordinate system. As a way, in this embodiment, (P _ex , _Pey , P _ez ) can be used to represent the first view coordinates. In this way, the viewing angle offset matrix T can be expressed as:

Alternatively, a transposed matrix N ^T may include a unit vector in the target area of the display space. As a way transposed matrix N ^T in the present embodiment may be expressed as:

Wherein, Vr, Vu, and Vn are the spatial unit vectors represented by the virtual image plane corresponding to the HUD display module. For example, please refer to FIG. 3, which shows an example diagram of the HUD virtual image plane of the HUD display device in this embodiment. As shown in Figure 3, V _r is the right vector, V _u is the upper vector, and V _n is the normal vector of the HUD virtual image plane.

Optionally, the projection matrix includes a field of view parameter, and the field of view may include a distance parameter and a scale parameter associated with the user's viewing angle. The relational expression satisfied by the projection matrix P can be:

Among them, the parameters n and f are the distance of the human eye's field of view (that is, the above distance parameters), and the parameters l, r, b, and t can represent the left, right, bottom, and top determined by the relationship between the size of the eye and the HUD virtual image plane and the pose. Scale (that is, the scale parameter mentioned above). Optionally, please refer to Fig. 4, which shows a schematic diagram of the relationship between the driver’s eyes and the HUD virtual image plane in this embodiment. As shown in Fig. 4, if d=-(v _n · v _a ), then the parameters can be obtained The calculation formulas of l, r, b, and t can be expressed as:

l=(v _r ·v _a )n/d r=(v _r ·v _b )n/d

b=(v _u ·v _a )n/d t=(v _u ·v _c )n/d

As a way, when the first transformation matrix and the second transformation matrix are obtained, the product of the parameter represented by the first transformation matrix and the parameter represented by the second transformation matrix can be obtained as real scene information and mapped to the target display The coordinate transformation rule corresponding to the area, that is, the coordinate transformation rule can be expressed as:

F=MC=MPN ^T T.

It should be noted that in this embodiment, the eye tracking device can capture the spatial pose of the driver's eyes in real time, and calculate the difference between the current moment and the previous moment in the driver's eye pose. Optionally, when the pose difference exceeds a preset specified threshold (the specific value may not be limited), the aforementioned second transformation matrix C may be recalculated and updated. Among them, the driver’s eye position difference can be calculated in a variety of ways, for example, it can be the mean square error MSE or the average absolute error MAE. It is understandable that the position of the driver’s eyes can also be customized according to actual needs. How the posture difference is calculated. Optionally, the designated thresholds corresponding to different calculation methods may be different.

Step S140: Generate driving guidance information based on the real scene information.

The driving guide information in this embodiment may include navigation instruction information corresponding to road conditions, pedestrian warning information, and tourist attractions prompt information, etc. The type and specific content of the driving guide information may not be limited. For example, as shown in FIG. 5, there is shown an example diagram of displaying driving guidance information through the augmented reality-based information display system proposed by this application in a dangerous scene provided by this embodiment. As shown in FIG. The real scene information collected by the scene perception device is converted into a HUD virtual image and displayed on the HUD display device. The specific content displayed is shown in the right image of Figure 5. In this case, the scene seen by the driver’s eyes may include lane guidance information ( That is, the "navigation instructions in the virtual image" shown in FIG. 5) and pedestrian warning information (ie, the "pedestrian prompt box in the virtual image" shown in FIG. 5).

As a way, when the real scene information is acquired, the driving guide information can be generated based on the real scene information. Optionally, the way of prompting the driving guide information in this embodiment may not be limited. Arrows), pictures, animations, voice or video, etc., then the driving guidance information for different prompts can be generated in a corresponding manner. Optionally, for the principle of generating driving guidance information based on actual scene information and each prompt mode, reference may be made to related technologies, which will not be repeated here. Optionally, the driving guide information in this embodiment may include at least one prompt method. For example, the navigation indicator icon corresponding to the road may be displayed in combination with voice prompting the user, so that the user can be driven more accurately. Guidance reminders to ensure driving safety and enhance user experience.

Step S150: Display the driving guidance information at the corresponding position of the target display area based on the coordinate transformation rule.

Optionally, by displaying the driving guide information at the corresponding position of the target display area based on the coordinate transformation rule, the difference between the position where the HUD displays the real-scene information and the actual position of the real-scene information can be avoided, and the accuracy and reliability of the display can be improved.

The present application provides an information display method based on augmented reality, which acquires real scene information collected by an image sensing device, and then acquires a target display area determined based on a user's first spatial pose. The first spatial pose is based on human eye tracking. The device determines, and then obtains the coordinate transformation rules corresponding to the real scene information mapped to the target display area, generates driving guidance information based on the real scene information, and then displays the driving guidance information in the corresponding position of the target display area based on the coordinate transformation rules. As a result, the driving guidance information generated based on the real scene information is displayed in the corresponding position of the target display area determined based on the user's first spatial pose determined by the eye tracking device through the coordinate transformation rule through the above method. The eye tracking device tracks the user's first spatial pose in real time, so that it can adapt to changes in the human eye pose, dynamically adjust the display area and display position of the driving guidance information, and realize that the user can accurately and conveniently drive Viewing the virtual driving guidance information corresponding to the driving scene improves the safety and comfort of driving, thereby enhancing the user experience.

Please refer to FIG. 6, which is a method flowchart of an augmented reality-based information display method provided by another embodiment of this application. The method of this embodiment may be executed by an augmented reality-based device for processing real-scene information, and the device may be implemented by hardware and/or software, and the method includes:

Step S210: Acquire real scene information collected by the scene sensing device.

Step S220: Obtain a target display area determined based on the user's first spatial pose.

Wherein, the first spatial pose is determined based on the eye tracking device, and the specific description can refer to the description in the foregoing embodiment, which will not be repeated here.

Step S230: Obtain a coordinate transformation rule corresponding to the mapping of the real scene information to the target display area.

Step S240: Generate driving guidance information based on the real scene information.

Step S250: Input the position coordinates of the real scene information in the coordinate system corresponding to the scene sensing device into the first transformation matrix to obtain the coordinate transformation matrix to be processed.

As a way, the position coordinates of the real scene information in the coordinate system corresponding to the scene sensing device may be input into the first transformation matrix, and the result obtained by the output may be used as the coordinate transformation matrix to be processed. For example, in a specific application scenario, it is assumed that the position coordinates of the actual scene information in the coordinate system corresponding to the scene sensing device are O _w (x, y, z). Optionally, the position coordinates O _w (x, y) , Z) After inputting the aforementioned first transformation matrix, we can get:

Among them, O'can be used as the coordinate transformation matrix to be processed, and O _W uses homogeneous coordinates.

Step S260: Perform coordinate transformation on the coordinate transformation matrix to be processed according to the second transformation matrix to obtain the relative position coordinates of the real scene information in the target display area.

As a way, the coordinate transformation matrix to be processed may be transformed according to the aforementioned second transformation matrix to obtain the relative position coordinates of the real scene information in the target display area. Optionally, the specific implementation process of coordinate transformation can refer to related technologies, which will not be repeated here.

For example, taking the above example as an example, assuming that in the HUD image of the target display area of the HUD display device, the position coordinate corresponding to the position coordinate O _w (x, y, z) is expressed as O _h (u, v), then After the coordinate transformation matrix O'to be processed is transformed according to the second transformation matrix, we can obtain:

Among them, width and height are the width and height of the HUD image, and the unit can be both pixels. In this way, _Oh (u, v) can be used as the relative position coordinates of the real scene information in the target display area.

Step S270: Display the driving guidance information at the position represented by the relative position coordinates.

Optionally, in this manner, the driving guide information corresponding to the real scene information may be displayed at the position represented by the relative position coordinates.

The following describes this embodiment with a specific example:

Please refer to FIG. 7, which shows an example diagram of the display effect of the information display system based on augmented reality provided by this embodiment. As shown in Figure 7, a virtual and real scene fusion system (which can be understood as an information display system based on augmented reality in this application) can be built in related modeling software (such as Unity3D, etc.). The virtual and real scene fusion system may include cars, Camera 1 (used to simulate the eyes of the driver), script program for simulating human eye tracking device 1, HUD imaging module simulated by camera 2 and plane together (i.e. the aforementioned HUD display device), image perception simulated by a checkerboard The spatial scene information (which can be spatial scene information in different scenes) acquired by the module (implemented by the script program 2), and the information transformation, image rendering and rendering of the image processing device completed by the script program 3.

Optionally, in the simulated virtual and real scene fusion system, the center position of the bottom of the car can be selected as the coordinate origin, the forward direction of the car is the positive direction of the Z axis, and the right-hand coordinate system is adopted. It is assumed that the driver is sitting in the driving position, and the camera 1 The simulated driver's eyes are facing forward, and the pose of the HUD virtual camera simulated by camera 2 is the same as that of the driver's eyes. In this way, the scene sensing device can obtain the checkerboard space corner information on the car , The image processing device can draw the corner point to the HUD image space (the lower left corner as shown in Figure 7 is the corner point image drawn by the image processing device to the HUD image space), and then send the drawn image to the HUD display device for processing Display (as shown in Figure 7 the corner image is sent to the HUD for virtual image display). In this way, the driver's perspective scene as shown in FIG. 7 can be obtained. Through the enlarged view of the virtual and real fusion result shown in FIG. 7, it can be seen that the virtual and real scenes can be accurately fused.

Wherein, the eye tracking device in this embodiment may be configured with a 3D sensor that is perceived by the spatial pose of the human eye and an eye tracking algorithm adapted to the output data of the sensor. Optionally, the 3D sensor for human eye spatial pose perception may be a binocular camera, or an RGB-D camera, etc., and the human eye tracking algorithm may be a computer vision algorithm, or a deep learning algorithm, etc., and the details are not limited. As a way, the scene sensing device can obtain real-life information in driving scenes such as lanes, navigation instructions, (dangerous) pedestrians, etc., through a camera combined with GPS, IMU sensors, and perception processing algorithms. Optionally, if the user's eye position changes, the image processing device can adjust the aforementioned first transformation matrix and the second transformation matrix according to the changed eye position, and then draw the same as the changed eye position. The image that matches the posture, and displays the virtual image information through the HUD display device.

As a way, if the spatial pose of the driver’s eyes changes during driving, for example, the driver’s line of sight moves as shown in Figure 5. In this case, the eye tracking device can be used in real time. Obtain the spatial pose of the driver's eyes, and determine whether the change between the human eye pose at the current moment and the human eye pose at the previous moment exceeds a specified threshold, where the value of the specified threshold can be set according to the actual situation. Optionally, if it is determined that the change of the driver’s eye pose exceeds a specified threshold, in this case, the driver’s spatial pose can be retrieved based on the changed driver’s eye pose, and then based on the spatial pose. The pose re-determines the target display area, and then displays the virtual and real fusion scene graph at the corresponding position of the re-determined target real area.

For example, as an embodiment, under the display effect shown in Figure 7, if the driver’s eye pose changes compared to the previous moment, and the range of the change exceeds the specified threshold, then you can get Figure 8 shows an example of the display effect. Wherein, the specific value of the time may not be limited, and in some possible implementation manners, the judgment interval may also be changed to a time period or a period. As shown in Figure 8, when the change range of the driver’s eye pose exceeds the specified threshold, the position where the virtual and real scene fusion map is displayed in the driver’s perspective scene is shifted, so that the driver’s eye position When the posture changes, the driving guidance information corresponding to the actual scene information in the driving scene can be presented to the user in the best perspective, to avoid driving safety accidents caused by the user's sight deviation, and to improve the safety and flexibility of driving. And then improve the user experience.

It should be noted that, in this embodiment, if the driver's eyes can still see the HUD virtual image plane after the sight of the driver moves, the driver can accurately see the navigation instructions marked on the driving lane or surround pedestrians through the HUD virtual image plane. Warning box and other driving guidance information to ensure safe driving. Optionally, if the driver’s eyes cannot see the HUD virtual image plane in front of the front of the vehicle after the driver’s line of sight moves, in some possible implementations, multiple HUD virtual image planes can be configured if necessary, so that the driver’s When the line of sight is shifted (or rotated) in any direction, you can see the driving guidance information that needs to be paid attention to in the current driving scene, which improves the flexibility and diversity of information display and enhances the user experience.

The present application provides an information display method based on augmented reality. The driving guidance information generated based on the real scene information is transformed by the first transformation matrix and the second transformation matrix respectively, and then displayed on the basis of the human eye tracking device. The determined first spatial pose of the user and then the determined corresponding position of the target display area realize the real-time tracking of the user's first spatial pose through the eye tracking device, so that it can adapt to the changes of the human eye pose and dynamically adjust The display area and display position of the driving guide information, so that the user can accurately and conveniently view the virtual driving guide information corresponding to the driving scene during driving, without the need to repeatedly confirm the accuracy of the driving guide information, reducing the need to check the road conditions As well as navigation and other driving guidance information, the frequent change of sight caused by fatigue has improved the safety and comfort of driving.

Please refer to FIG. 9, which is a method flowchart of an augmented reality-based information display method provided by another embodiment of this application. The method of this embodiment may be executed by an augmented reality-based device for processing real-scene information, and the device may be implemented by hardware and/or software, and the method includes:

Step S310: Acquire the real scene information collected by the scene sensing device.

Step S320: Obtain a target display area determined based on a first spatial pose of the user, where the first spatial pose is determined based on the eye tracking device.

Step S330: Detect the change of the first spatial pose by acquiring the eye posture change parameter of the eye tracking device.

It is understandable that, during the user's driving process, the user's posture may change according to the driving scene information such as the road conditions of the traveling vehicle. For example, the user's head will rotate in the direction In this mode, the user's line of sight will change (for example, the line of sight is shifted, etc.). In this case, if the original HUD display method is still used to display the driving guidance information corresponding to the actual scene information, there may be display errors due to the position Cause safety hazards.

As a way to improve this problem, the eye tracking device in this embodiment can detect the user's eye posture in real time. If the human eye pose changes, then the parameters corresponding to the human eye pose change can be obtained. Optionally, in this way, the user’s first position can be detected through the human eye pose change (ie, the human eye pose change) parameter. A change in spatial pose so that if a change in the first spatial pose is detected, the target display area will be re-determined based on the changed spatial pose, so as to ensure the accuracy of the display position of the driving guidance information corresponding to the actual scene information Without requiring the user to repeatedly confirm the accuracy of the driving guidance information, the flexibility of displaying the driving guidance information is improved, thereby enhancing the user experience.

Optionally, the human eye posture change parameter may include the direction, angle, or range of the human eye's line of sight. As a way, the relevant face recognition algorithm can be used to determine whether the eye pose has changed in the eye pose image collected by the eye tracking device. If there is a change, the corresponding eye pose change parameter can be obtained .

As a way, the change vector corresponding to the first spatial pose can be obtained according to the human eye pose change parameter. Optionally, the specific calculation process can be implemented with reference to related technologies, which will not be repeated here.

Step S340: Obtain a coordinate transformation rule corresponding to the mapping of the real scene information to the target display area.

Step S350: Generate driving guidance information based on the real scene information.

Step S361: If the amount of change of the human eye posture change parameter is greater than a preset threshold, update the coordinate transformation rule according to the human eye posture change parameter.

Optionally, a preset threshold corresponding to the change vector may be configured in advance, and the preset threshold may be used to distinguish whether subsequent adjustments to the target display area are required. As an implementation manner, if the change value of the change vector is greater than the preset threshold, the target display area can be adjusted based on the change vector to obtain the newly determined target display area.

For example, in a specific application scenario, please refer to FIG. 10, which shows an example diagram of adjusting the target display area based on the change vector provided in this embodiment. As shown in Figure 10, the user's first spatial pose has changed from 22 to 23', where 23' is the current first spatial pose, which is determined based on the user's current human eye pose . As a way, if it is detected that the change vector of the human eye pose corresponding to the user's current first spatial pose 23' is greater than the preset threshold, in this way, it is located on the screen 21 of the front windshield of the car The position of the target display area can be changed from 23 to 23', where 23' is the re-determined target display area.

Optionally, if the user's first spatial pose changes, in the above manner, the coordinate transformation rules can be updated according to the human eye pose change parameters to obtain the real scene information mapped to the second coordinate corresponding to the newly determined target display area Conversion rules, where the specific determination process of the second coordinate conversion rule can refer to the determination principle and determination process of the aforementioned coordinate conversion rule, which will not be repeated here.

Step S362: Display the driving guidance information at the corresponding position of the target display area based on the updated coordinate transformation rule.

Optionally, on the basis of acquiring the newly determined target display area, the driving guidance information may be displayed at the corresponding position of the newly determined target display area based on the second coordinate change rule.

As an implementation manner, the target display area in this embodiment can be adjusted according to the change of the user's first spatial pose. For example, if it is detected that the user has a gesture such as bowing, the target display area can be displayed in the corresponding position on the central control display; if it is detected that the user looks at the phone more frequently during driving, the target display area can be displayed to On the display screen of the mobile phone; or other screens that can be used as the target display area in the driving scene, for example, the windows on the left and right sides of the driving position.

Step S371: If the amount of change of the human eye posture change parameter is not greater than the preset threshold, display the driving guidance information at the corresponding position of the target display area based on the coordinate transformation rule.

As another implementation manner, if the change value of the change vector is not greater than the preset threshold, the target display area determined based on the user's first spatial pose can be obtained. In this way, the user's first spatial pose can be For the sitting posture of the user, please refer to the description in the foregoing embodiment for details.

Optionally, in this embodiment, the sequence of the various steps may not be limited. For example, step S330 may be implemented after step S340.

Exemplarily, a specific implementation process is shown below:

As shown in FIG. 11, an example diagram of the processing procedure of the information display method based on enhanced display proposed in this embodiment is shown. In FIG. 11, the process pointed by the hollow arrow may be the initial process, and the process pointed by the solid arrow may be the real-time continuous process. As an implementation manner, the coordinate system can be established first, and then the space pose of the scene perception module (which can be understood as the aforementioned scene perception device) and the HUD virtual image plane are measured, and the driver's eye space position coordinates are initialized, and then calculated separately Scene perception module matrix M and HUD imaging matrix C, and then the total transformation matrix (that is, the aforementioned coordinate transformation rule) F=CM is obtained. Optionally, the scene perception module can acquire real-time information in real time, use the real-time information as the information to be displayed and send it to the image processing module (which can be understood as the aforementioned image processing device), and the image processing module performs coordinate change processing on the coordinates corresponding to the real-world information The final image is drawn, and the image is projected onto the HUD display screen (that is, the aforementioned target display area) for display, so as to improve the accuracy of the display position of the driving guide information, reduce user operations, and improve user experience.

Step S364: Display the driving guidance information at the corresponding position of the target display area based on the coordinate transformation rule.

The present application provides an information display method based on augmented reality, which detects the change of the first spatial pose by acquiring the eye posture change parameter of the eye tracking device, and then the change value of the change vector corresponding to the eye posture change parameter is greater than In the case of a preset threshold, the target display area is re-adjusted based on the change vector, and the driving guidance information is displayed in the corresponding position of the target display area determined based on the user's first spatial pose determined based on the eye tracking device. In order to track the user’s first spatial posture in real time through the eye tracking device, it can adapt to changes in the eye’s posture, dynamically adjust the display area and display position of the driving guidance information, so that the user can be accurate during driving. Conveniently view the virtual driving guidance information corresponding to the driving scene without repeatedly confirming the accuracy of the driving guidance information. This reduces the fatigue of frequent sight changes caused by viewing road conditions and driving guidance information such as navigation, and improves driving safety and Comfort.

Referring to FIG. 12, an information display device 400 based on augmented reality provided by an embodiment of the present application can be run on a projection device. The device 400 includes:

The image sensing module 410 is used to obtain real scene information collected by the image sensing device.

The coordinate transformation module 420 is configured to obtain the target display area determined based on the user's first spatial pose, which is determined based on the human eye tracking device.

Optionally, the device 400 may further include a parameter change detection module, which is used to detect the change of the first spatial pose by acquiring the eye posture change parameter of the eye tracking device. In this manner, the coordinate transformation module 420 may be specifically used to obtain the eye posture change parameters collected by the eye tracking device; obtain the change vector corresponding to the first spatial pose based on the eye posture change parameters If the amount of change of the change vector is greater than a preset threshold, adjust the target display area based on the change vector to obtain a re-determined target display area. Optionally, if the amount of change of the change vector is not greater than a preset threshold, the step of obtaining the target display area determined based on the user's first spatial pose is performed.

As a way, the coordinate transformation module 420 may also be used to obtain a coordinate transformation rule corresponding to the real scene information mapped to the target display area.

Optionally, the coordinate transformation rule in this embodiment may include a first transformation matrix and a second transformation matrix. The first transformation matrix is used to determine the reference world coordinates corresponding to the coordinates of the real scene information collected by the scene sensing device. The second transformation matrix is used to convert the reference world coordinates into view coordinates in the target display area that match the offset of the user's viewing angle.

Optionally, the first transformation matrix may include a first rotation matrix and a first translation vector, where the first rotation matrix is used to rotate the coordinates of the real scene information collected by the scene sensing device, and the first translation vector is used To translate the coordinates, the first transformation matrix determines the reference world coordinates corresponding to the coordinates of the real scene information collected by the scene sensing device based on the first rotation matrix and the first translation vector.

Optionally, the second transformation matrix may include a viewing angle offset matrix, a transposition matrix, and a projection matrix. The projection matrix is used to determine a mapping range for mapping the real scene information to the target display area, and the viewing angle offset The matrix is used to determine the degree of deviation of the user’s viewing angle detected by the eye tracking device, the transposed matrix is used to determine the relative position within the mapping range where the driving guidance information is displayed, and the second The transformation matrix converts the reference world coordinates into view coordinates in the target display area that match the offset of the user's view angle based on the view angle offset matrix, the transpose matrix, and the projection matrix.

Optionally, the viewing angle offset matrix may include first view coordinates, the transposition matrix represents a transposition facing the first transformation matrix, and the transposition matrix includes a spatial unit vector located in the target display area, so The projection matrix includes a field angle parameter, and the field angle parameter includes a distance parameter and a scale parameter associated with the user's viewing angle.

The display module 430 is configured to generate driving guidance information based on the real scene information.

As a way, the display module 430 may also be used to display the driving guidance information at the corresponding position of the target display area based on the coordinate transformation rule.

Optionally, the display module 430 may be specifically configured to input the position coordinates of the real scene information in the coordinate system corresponding to the scene sensing device into the first transformation matrix to obtain the coordinate transformation matrix to be processed; The coordinate transformation matrix performs coordinate transformation according to the second transformation matrix to obtain the relative position coordinates of the real scene information in the target display area; the driving guidance information is displayed at the position characterized by the relative position coordinates.

It should be noted that the device embodiment in this application and the foregoing method embodiment correspond to each other, and the specific principles in the device embodiment can be referred to the content in the foregoing method embodiment, which will not be repeated here.

Hereinafter, a projection device provided by the present application will be described with reference to FIG. 13.

Please refer to FIG. 13, based on the foregoing augmented reality-based information display method, system, and device, an embodiment of the present application also provides another projection device 100 that can execute the foregoing augmented reality-based information display method. The projection device 100 includes one or more (only one shown in the figure) processor 102, memory 104, image perception module 11, coordinate transformation module 12, human eye tracking module 14 and display module 13 coupled with each other. Wherein, the memory 104 stores a program that can execute the content in the foregoing embodiment, and the processor 102 can execute the program stored in the memory 104, and the memory 104 includes the device 400 described in the foregoing embodiment.

The processor 102 may include one or more processing cores. The processor 102 uses various interfaces and lines to connect various parts of the entire projection device 100, and executes by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and calling data stored in the memory 104. Various functions and processing data of the projection device 100. Optionally, the processor 102 may use at least one of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). A kind of hardware form to realize. The processor 102 may integrate one or a combination of a central processing unit (CPU), a video image processor (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is used for rendering and drawing of display content; the modem is used for processing wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 102, but may be implemented by a communication chip alone.

The memory 104 may include random access memory (RAM) or read-only memory (Read-Only Memory). The memory 104 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 104 may include a storage program area and a storage data area, where the storage program area may store instructions for implementing the operating system and instructions for implementing at least one function (such as touch function, sound playback function, video image playback function, etc.) ), instructions used to implement the foregoing method embodiments, etc. The data storage area can also store data (for example, audio and video data) created by the projection device 100 during use.

The image perception module 11 is used to obtain the real scene information collected by the image perception device; the coordinate transformation module 12 is used to obtain the target display area determined based on the user's first spatial pose, which is based on the human eye tracking device 14 The eye tracking device 14 is used to detect the user's eye pose in real time; the coordinate transformation module 12 is also used to obtain the coordinate transformation rule corresponding to the real scene information mapped to the target display area; the display The module 13 is configured to generate driving guidance information based on the real scene information; the display module 13 is also configured to display the driving guidance information in a corresponding position of the target display area based on the coordinate transformation rule.

Please refer to FIG. 14, which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. The computer-readable medium 500 stores program code, and the program code can be invoked by a processor to execute the method described in the foregoing method embodiment.

The computer-readable storage medium 500 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 500 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 500 has a storage space for the program code 510 for executing any method steps in the above-mentioned methods. These program codes can be read from or written into one or more computer program products. The program code 510 may be compressed in a suitable form, for example.

In summary, the present application provides an augmented reality-based information display method, system, device, projection equipment, and storage medium. By acquiring the real scene information collected by the image sensing device, the first spatial pose determination based on the user is obtained. The first spatial pose is determined based on the human eye tracking device, and then the real scene information is mapped to the coordinate transformation rule corresponding to the target display area, and the driving guidance information is generated based on the real scene information, and then the driving guidance is based on the coordinate transformation rule The information is displayed in the corresponding position of the target display area. As a result, the driving guidance information generated based on the real scene information is displayed in the corresponding position of the target display area determined based on the user's first spatial pose determined by the eye tracking device through the coordinate transformation rule through the above method. The eye tracking device tracks the user's first spatial pose in real time, so that it can adapt to changes in the human eye pose, dynamically adjust the display area and display position of the driving guidance information, and realize that the user can accurately and conveniently drive Viewing the virtual driving guidance information corresponding to the driving scene improves the safety and comfort of driving, thereby enhancing the user experience.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the foregoing embodiments are modified, or some of the technical features thereof are equivalently replaced; these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

An information display method based on augmented reality, characterized in that the method includes:

Obtain real scene information collected by the scene sensing device;

Acquiring a target display area determined based on a first spatial pose of the user, where the first spatial pose is determined based on a human eye tracking device;

Acquiring a coordinate transformation rule corresponding to the mapping of the real scene information to the target display area;

Generating driving guidance information based on the real scene information;

Displaying the driving guidance information in a corresponding position of the target display area based on the coordinate transformation rule.
The method according to claim 1, wherein the coordinate transformation rule includes a first transformation matrix and a second transformation matrix, and the first transformation matrix is used to determine the coordinates of the real scene information collected by the scene sensing device. Corresponding to the reference world coordinates, the second transformation matrix is used to convert the reference world coordinates into view coordinates in the target display area that match the offset of the user's viewing angle.
2. The method according to claim 2, wherein the first transformation matrix comprises a first rotation matrix and a first translation vector, and the first rotation matrix is used to determine the coordinates of the real scene information collected by the scene sensing device. Performing a rotation, the first translation vector is used to translate the coordinates, and the first transformation matrix is based on the first rotation matrix and the first translation vector to determine the difference between the real scene information collected by the scene sensing device and the The coordinate corresponds to the reference world coordinate.
The method according to claim 2, wherein the second transformation matrix comprises a viewing angle offset matrix, a transposition matrix, and a projection matrix, and the projection matrix is used to determine the mapping of the real scene information to the target display The mapping range of the region, the viewing angle offset matrix is used to determine the degree of deviation of the user's viewing angle detected by the eye tracking device, and the transposed matrix is used to determine that the driving is displayed within the mapping range. The relative position of the guide information, the second transformation matrix converts the reference world coordinates into an offset match with the user’s viewing angle in the target display area based on the viewing angle offset matrix, the transposed matrix, and the projection matrix The view coordinates.
The method according to claim 4, wherein the viewing angle offset matrix includes first view coordinates, the transposition matrix represents a transposition facing the first transformation matrix, and the transposition matrix includes The spatial unit vector of the target display area, the projection matrix includes a field of view parameter, and the field of view parameter includes a distance parameter and a scale parameter associated with the viewing angle of the user.
The method according to any one of claims 2-5, wherein the displaying the driving guidance information at a corresponding position of the target display area based on the coordinate transformation rule comprises:

Inputting the position coordinates of the real scene information in the coordinate system corresponding to the scene sensing device into the first transformation matrix to obtain the coordinate transformation matrix to be processed;

Performing coordinate transformation on the coordinate transformation matrix to be processed according to the second transformation matrix to obtain the relative position coordinates of the real scene information in the target display area;

The driving guidance information is displayed at the position represented by the relative position coordinates.
The method according to claim 1, characterized in that, before acquiring the real scene information and mapping the coordinate transformation rule corresponding to the target display area, the method further comprises:

Detecting the change in the first spatial pose by acquiring the eye posture change parameter of the eye tracking device;

The method also includes:

If the amount of change of the human eye posture change parameter is greater than a preset threshold, the coordinate transformation rule is updated according to the human eye posture change parameter.
The method according to claim 7, wherein the method further comprises:

If the amount of change of the human eye posture change parameter is not greater than a preset threshold, the step of acquiring the coordinate transformation rule corresponding to the mapping of the real scene information to the target display area is performed.
An information display device based on augmented reality, characterized in that the information display device includes an image perception module, a coordinate transformation module, and a display module:

The image sensing module is used to obtain real-life information collected by the image sensing device;

The coordinate transformation module is configured to obtain a target display area determined based on a user's first spatial pose, where the first spatial pose is determined based on a human eye tracking device;

The coordinate transformation module is also used to obtain the coordinate transformation rule corresponding to the real scene information mapped to the target display area;

The display module is used to generate driving guidance information based on the real scene information;

The display module is further configured to display the driving guidance information in a corresponding position of the target display area based on the coordinate transformation rule.
A vehicle-mounted information display system based on augmented reality is characterized in that the system includes:

Scene perception device, used to collect real scene information of the vehicle's external environment;

Eye tracking device, used to obtain the user's line of sight movement range;

An image processing device for acquiring a first spatial pose of the user based on the movement range of the line of sight, acquiring a target display area determined based on the first spatial pose, and acquiring the real scene information mapped to the target display area A corresponding coordinate transformation rule, generating driving guidance information based on the real scene information, and generating the target position coordinates of the driving guidance information displayed in the target display area based on the coordinate transformation rule;

The HUD display device is used to display the driving guidance information at the target position coordinates of the target display area.
A projection device, characterized in that it comprises one or more processors and a memory;

One or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute any one of claims 1-8 method.
A computer-readable storage medium, wherein a program code is stored in the computer-readable storage medium, wherein the method according to any one of claims 1-8 is executed when the program code is run by a processor.