WO2021218602A1

WO2021218602A1 - Vision-based adaptive ar-hud brightness adjustment method

Info

Publication number: WO2021218602A1
Application number: PCT/CN2021/086415
Authority: WO
Inventors: 余新; 邓岳慈; 康瑞
Original assignee: 深圳光峰科技股份有限公司
Priority date: 2020-04-29
Filing date: 2021-04-12
Publication date: 2021-11-04
Also published as: CN113573035A

Abstract

Disclosed is a vision-based adaptive AR-HUD brightness adjustment method. The adjustment method comprises: acquiring a scene image in front of a vehicle window; generating an HUD display image according to the scene image, and acquiring the brightness distribution of the HUD display image; dividing the HUD display image into one or more image regions; calculating, according to the brightness distribution of the scene image, a target brightness corresponding to each of the image regions; and adjusting, according to the target brightness corresponding to the image region, a light source power corresponding to the image region. In the vision-based adaptive AR-HUD brightness adjustment method of the present invention, the intensity of ambient background light is quantitatively calculated by means of a visual perception method, a brightness-light source power mapping relationship is established according to the range of a comfortable brightness perceived by human eyes, and the brightness of an image finally displayed by an HUD is adjusted, so as to realize the dynamic adjustment of an HUD image brightness, such that not only can the overall perception and experience of a driver be improved, the overall power of an HUD is also reduced, thereby alleviating the energy consumption and heat dissipation pressure.

Description

A vision-based adaptive adjustment method of AR-HUD brightness

Technical field

The present invention relates to the technical field of HUD brightness adjustment, in particular to a vision-based AR-HUD brightness adaptive adjustment method.

Background technique

AR-HUD is a technology that integrates Augmented Reality into a head-up display to solve the problem of lack of traditional HUD content and no immersive experience. The HUD uses a projector to project the displayed image on the windshield. After optical imaging, a virtual image can be displayed a few meters away in front of the car window. The on-board computing system renders the important information needed by the driver in the virtual screen through calculation, such as navigation information and road condition information, and the rendered information can be integrated with the real scene in the driver's field of vision to achieve the effect of AR . However, in a real application scenario, due to the objective influence of ambient light and scene objects, the HUD display effect of the same brightness will be greatly reduced. For example, under strong background light, the display brightness needs to be enhanced, but the virtual image range at this time If there is an area with weak background light, the excessively strong display brightness will bring discomfort to the driver. Adaptively adjusting the brightness of the light source displayed by the HUD can not only improve the overall experience of the driver, but also reduce the overall power of the HUD, alleviating energy consumption and heat dissipation pressure.

There are currently two schemes for adjusting the brightness of the HUD, one is manual adjustment, and the other is based on a photoelectric sensor. Manual adjustment requires human eyes to judge the comfort level of the screen display brightness, which is inconvenient to adjust during driving. The photoelectric sensor-based method relies heavily on the accuracy of the sensor and the effective range of the sensor, and usually requires multiple sensors to be installed on the vehicle, which is very inconvenient to deploy. In addition, the above two methods can only adjust the global light source power of the HUD projection, and cannot adjust the brightness of the displayed picture locally. If the ambient background light of the displayed picture is unevenly distributed in space, then the adjusted image brightness is in the human eye. When you feel it, there will be a bright and no bright situation, which will affect the driver's experience.

Summary of the invention

The technical problem mainly solved by the present invention is to provide a vision-based AR-HUD brightness adaptive adjustment method, which solves the current technical problems of inconvenient adjustment of the brightness of the HUD and the inability to adjust the brightness locally.

In order to solve the above technical problems, a technical solution adopted by the present invention is to provide a vision-based AR-HUD brightness adaptive adjustment method, the adjustment method includes:

Obtain the scene image in front of the car window;

Generating a HUD display image according to the scene image, and acquiring the brightness distribution of the HUD display image;

Dividing the HUD display image into one or more image regions;

Calculating the target brightness corresponding to each of the image regions according to the brightness distribution of the scene image;

The power of the light source corresponding to the image area is adjusted according to the target brightness corresponding to the image area.

Wherein, the generating a HUD display image according to the scene image includes:

The scene image in front of the car window is captured, an RGB image and a depth image are obtained according to the scene image, and a HUD display image is generated from the RGB image and the depth image.

Wherein, the acquiring the brightness distribution of the HUD display image includes:

The depth image is converted into a 3D point cloud, and the 3D point cloud is mapped to the HUD display image to obtain a brightness distribution.

Wherein, the dividing the HUD display image into one or more image regions includes:

Use the entire HUD display image as an image area; or

According to the brightness value in the brightness distribution, the HUD display image is divided into more than one area.

Wherein, the dividing the HUD display image into more than one area according to the brightness value in the brightness distribution includes:

The edge detection algorithm is used to extract the edge information of the brightness distribution, and the edge information is fitted with a rectangular frame to obtain several image regions.

Wherein, the calculating the target brightness corresponding to each of the image regions according to the brightness distribution of the scene image includes:

The average brightness value of each image area is calculated, and the target brightness is calculated according to the average brightness value.

Wherein, the adjusting the light source power corresponding to the image area according to the target brightness corresponding to the image area includes:

Obtain preset brightness-light source power mapping data, obtain the light source power reference value of different image areas according to the target brightness value of each image area, and use the light source power reference value to drive the light-emitting elements of the corresponding image area of the HUD light source.

Wherein, the conversion formula for converting the depth image into a 3D point cloud is:

Wherein, the z _c is the depth value of the depth image pixel, d _x and d _y are the actual size of the pixel in the photosensitive chip of the camera that shoots the depth image, u _o and v _o are the center of the camera image plane, f is the focal length of the camera. d _x , d _y , u _o , v _o and f are the internal parameters of the camera, and the 3×4 RT matrix is the external parameters of the camera.

Wherein, the conversion formula for mapping the 3D point cloud to the image displayed by the HUD is:

Where d _x , d _y , u _o , v _o and f are the internal parameters of the camera, the 3×4 RT matrix is the external parameters of the camera, and X _w , Y _w , and Z _w are the 3D points Cloud parameters.

To solve the technical problem, the present invention also provides a computer-readable storage medium, the storage medium stores a computer program, and when the program is executed, it is used to implement the steps of the above-mentioned method.

Compared with the prior art, the beneficial effect of the vision-based AR-HUD brightness adaptive adjustment method of the present invention is that the intensity of the ambient background light is quantified through the method of visual perception, and the comfortable brightness range felt by the human eye is established. Brightness-light source power mapping relationship, adjust the HUD final display image brightness, realize the dynamic adjustment of HUD image brightness, not only can improve the overall experience of the driver, but also reduce the overall power of the HUD, relieve energy consumption and heat dissipation pressure.

Description of the drawings

In order to explain the technical solutions in the embodiments of the present invention more clearly, the following will briefly introduce the drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

Fig. 1 is a flowchart of a vision-based AR-HUD brightness adaptive adjustment method of the present invention;

Figure 2 is a specific flow chart of step S200;

Figure 3 is a specific flow chart of step S300;

Figure 4 is a schematic diagram of HUD brightness distribution;

Figure 5 is a schematic diagram of a vision-based AR-HUD brightness adaptive adjustment system;

Fig. 6 is a distribution diagram of a vision-based AR-HUD brightness adaptive adjustment system in a car.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific sequence. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes unlisted steps or units, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment.

The reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

The present invention provides a vision-based AR-HUD brightness adaptive adjustment method. The adjustment method includes: obtaining a scene image in front of a car window; generating a HUD display image according to the scene image, obtaining the brightness distribution of the HUD display image; displaying the HUD image Divide into one or more image areas; calculate the target brightness corresponding to each image area according to the brightness distribution of the scene image; adjust the light source power corresponding to the image area according to the target brightness corresponding to the image area. Through the method of visual perception, the intensity of the ambient background light is quantified and calculated, and the brightness-light source power mapping relationship is established according to the comfortable brightness range experienced by the human eye, and the final image brightness displayed by the HUD is adjusted to realize the dynamic adjustment of the HUD image brightness. A detailed description will be given below through specific embodiments.

Please refer to FIG. 1. FIG. 1 is a flowchart of a vision-based AR-HUD brightness adaptive adjustment method of the present invention. The adjustment method in FIG. 1 is implemented by step S100, step S200, step S300, step S400, and step S500, where,

Step S100 is to obtain a scene image in front of the car window. The scene image in front of the car window includes the external scene in front of the car window and the lighting conditions, which are used to adjust the brightness of the HUD to form an adjustment benchmark. The brightness adjustment of the HUD uses the scene image information in front of the car window as the contrast object.

There are many ways to obtain the scene image in front of the car window, for example, by setting up a photoelectric sensor, a camera or a camera with a capture program near the car window, and by sensing or capturing the scene image in front of the car window. The brightness in front of the car window.

Step S200 is to generate a HUD display image according to the scene image, and obtain the brightness distribution of the HUD display image. The method for acquiring the scene image in this embodiment is to set a camera in front of the car window to shoot the scene image of the outside world to obtain an RGB image (red, green, and blue primary color image) or an RGB image and a depth image. Image or RGB image and depth image conversion to get the brightness in front of the car window.

Example one:

The first embodiment is to set up a monocular camera in front of the car window. The monocular camera is formed based on a monocular hand-eye camera and a laser rangefinder. It uses a monocular with a line laser, a single CCD camera, pinhole imaging, and a laser surface constraint model. The visual measurement method takes pictures of the external environment to generate RGB images. After obtaining the RGB image of the scene outside the window of the car in the process of traveling, the RGB image is converted into a YUV image, and the value of the Y channel in the YUV image is extracted. The conversion formula is:

Y=0.299R+0.587G+0.114B,

U=-0.147R-0.289G-0.436B,

V＝0.615R-0.515G-0.100B,

Among them, R, G, and B are the values of red, green, and blue in the RGB image, respectively. The global average value of the Y channel under the YUV image is calculated by the above conversion formula, so as to obtain the brightness of the scene outside the car window.

Embodiment two:

The second embodiment is to set up an RGBD camera in front of the car window to obtain the RGB image and the depth image of the scene outside the car window when the car is traveling. In this embodiment, the RGBD camera is a binocular camera, and two cameras fixed at different positions are used for positioning. For example, two RGBD cameras are installed on both sides of a car window, so that the obtained image covers the entire car window. Part of the scene, the image of the external environment on the two camera image planes were obtained, RGB image and depth image were generated, and then the RGB image and depth image were converted and mapped to obtain the HUD display image.

Please refer to Figure 2. Figure 2 is a specific flowchart of step S200. Step S200 further includes step S201. Step S201 is to generate a HUD display image according to the scene image. RGB image and depth image, HUD display image is generated from RGB image and depth image.

Step S201 uses the second embodiment described above to generate a HUD display image, and an RGBD camera is set near the front windshield of the car. The RGBD camera shoots to form an RGB image and a depth image, and then the RGB image and the depth image are subsequently converted and mapped to generate a HUD Display the image. The RGBD camera in this embodiment is a binocular camera, the depth image is calculated from the RGB image obtained by the RGBD camera, and the pixels of the RGB image and the depth image are in one-to-one correspondence. The RGB image selected when calculating the depth image is any one of the RGB images obtained by the two left and right RGBD cameras. Step S200 further includes step S202. In step S202, obtaining the brightness distribution of the HUD display image includes: converting the depth image into a 3D point cloud, and mapping the 3D point cloud to the HUD display image to obtain the brightness distribution. The color image is acquired by the camera, and then the color information of the pixel at the corresponding position is assigned to the corresponding point in the point cloud, which is recorded in the form of points. Each point contains three-dimensional coordinates and color information, that is, the depth image is converted into a 3D point cloud. Then map the 3D point cloud to the HUD display image, the HUD display image is accompanied by color information, and the brightness distribution can be obtained.

The conversion formula for converting the depth image into a 3D point cloud in this embodiment is:

Among them, z _c is the depth value of the depth image pixel, d _x and d _y are the actual size of the pixel in the camera's photosensitive chip that shoots the depth image, u _o and v _o are the center of the camera image plane, and f is the focal length of the camera. d _x , d _y , u _o , v _o and f are the internal parameters of the camera, and the 3×4 RT matrix is the external parameters of the camera.

In projection geometry, spatial coordinates are often expressed in homogeneous coordinates, so the internal and external parameters of the camera are extended to 4×3 matrices and 4×4 matrices. The internal and external parameters of the camera are obtained in advance through calibration, and the obtained point cloud information includes spatial coordinates x _w , y _w , z _w and a brightness intensity value. The conversion formula for mapping the 3D point cloud to the image displayed by the HUD in this embodiment is:

Among them, d _x , _dy , u _o , v _o and f are the internal parameters of the camera, the 3×4 RT matrix is the external parameters of the camera, and x _w , y _w , and z _w are the parameters of the 3D point cloud.

Each 3D point cloud carries a piece of brightness information, so the HUD image after 3D point cloud mapping is the brightness distribution map, which contains the brightness information of the external scene image.

Step S300 is to divide the HUD display image into one or more image regions, and determine the brightness adjustment of different regions by dividing different image regions, so as to make the user's visual experience more comfortable.

Please refer to FIG. 3, which is a specific flow chart of step S300. Step S300 further includes steps S301 and S302. In step S301, the HUD display image is regarded as an image area. In step S302, the HUD display image is displayed according to the brightness value in the brightness distribution. Divide into more than one area.

If step S301 is executed, the entire HUD display image is regarded as an image area, so that the overall brightness of the HUD display image is adjusted. The adjustment method of step S301 adopts the first embodiment described above to generate a HUD display image, set a monocular camera to photograph the external environment, generate an RGB image, then convert the RGB image into a YUV image, and extract the global average value of the Y channel in the YUV image , The calculation formula is:

Among them, y _i is the size of the pixels of the Y channel of the image, and N is the number of pixels in a single channel of the image. Before the brightness adjustment, the brightness-light source power mapping data is preset, the light source power reference value is obtained according to the calculated global average value of the Y channel, and the light-emitting element of the HUD light source is driven by the light source power reference value. The brightness-light source power mapping data is established through experimental data in advance to form a corresponding functional relationship.

If step S302 is executed, the HUD display image is divided into a plurality of image areas, so that the brightness of different HUD display areas is adjusted respectively, and the division method is based on the brightness value in the brightness distribution. In step S202, the depth image is converted into a 3D point cloud, and the 3D point cloud is mapped to the HUD display image to obtain the brightness distribution, and the brightness value is obtained from the brightness distribution. There are at least two ways to obtain the brightness value. One of them is to cluster adjacent pixels according to the brightness value of each point in the brightness distribution. For example, the K-means algorithm is used to center k points in the space. Clustering, classifying the objects closest to them, averaging all the data points belonging to this class, taking the average as the new class center, repeating until convergence, to get the brightness value of each point in the brightness distribution. Another acquisition method is to divide the HUD display image into multiple regions in advance, merge the adjacent regions whose average brightness difference is less than the threshold in the multiple regions, and then calculate the brightness value of each region.

The division of the HUD display image into more than one region according to the brightness value in the brightness distribution in this application includes: using an edge detection algorithm to extract edge information of the brightness distribution, and fitting the edge information with a rectangular frame to obtain several image regions. Please refer to Figure 4. Figure 4 shows the brightness distribution of the HUD at a certain moment, and the edge information is calculated by the edge detection algorithm (such as the Canny edge detection algorithm). The dashed line is the sub-region fitted by the rectangle. If there is an intersection region, the average brightness of the region is the average value of the average brightness of the two intersecting regions. The brightness value corresponding to each area and the pixel coordinate set of the area will be stored.

Step S400 in the present application is to calculate the target brightness corresponding to each image area according to the brightness distribution of the scene image, specifically, calculating the average brightness value of each image area, and calculating the target brightness according to the average brightness value. If the entire HUD display image is regarded as an image area, the average brightness value of the entire image is calculated. If the HUD display image is divided into multiple regions, the average brightness value of each region is calculated separately. According to the brightness of each area of the scene image, the target brightness that best meets the user's visual experience is calculated.

Step S500 in the present application is to adjust the light source power corresponding to the image area according to the target brightness corresponding to the image area, and adjust the light source power to change the brightness of the image area so that the brightness value is consistent with the target brightness.

Specifically, the preset brightness-light source power mapping data is obtained, the light source power reference value of different image areas is obtained according to the target brightness value of each image area, and the light-emitting elements of the corresponding image area of the HUD light source are driven by the light source power reference value. This application establishes brightness-light source power mapping data through experimental data in advance, and selects multiple sets of different brightness and light source power data through multiple experiments, such as connecting the experimental results to select points to form the corresponding relationship between brightness and light source power, and then Fit the experimental data to form a functional relationship. When the brightness needs to be adjusted, the corresponding light source power is determined according to the above functional relationship, and the corresponding light source power is used to drive the light-emitting elements of the corresponding image area of the HUD light source.

In order to solve the technical problem, the present invention also provides a computer-readable storage medium, the storage medium stores a computer program, and when the program is executed, it is used to implement the steps of the above adjustment method. The aforementioned computer-readable storage medium may be any electronic, magnetic, optical, or other physical storage device, and may contain or store information, such as executable instructions, data, and so on. For example, the computer-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard drive), any type of storage disk (such as optical disk) , DVD, etc.), or similar storage media, or a combination of them.

The present invention also provides a vision-based AR-HUD brightness adaptive adjustment system. Please refer to Figures 5 and 6. The system includes a visual perception module 100, a display control module 200, a HUD display module 300, and a storage module 400:

Among them, the visual perception module 100 captures external environment information, that is, the scene image in front of the car window, and sends the scene image to the display control module 200.

The display control module 200 receives the external environment information sent by the visual perception module 100, evaluates the brightness of the external environment, and outputs the light source power signal displayed by the HUD according to the brightness.

The HUD display module 300 receives the output signal of the display control module 200, and projects the HUD display image with the light source power of the output signal.

The storage module 400 is used to store brightness and light source power mapping data; the display control module obtains the corresponding light source power according to the external environment information sent by the visual perception module and the brightness and light source power mapping data in the storage module, and sends it to the HUD display module. The brightness and light source power are mapped as a functional relationship established in advance through experimental data. The experimental object is the corresponding relationship between the car's light source power and the brightness. The corresponding relationship between the brightness and the light source power is formed through a certain method, such as the connection of the selected points in the experiment. Import the functional relationship formed by the experimental data into the storage module for subsequent adjustment of the light source power accordingly.

The present invention also provides a vision-based AR-HUD brightness adaptive adjustment device. The adjustment device includes a HUD display device, a controller, and a monocular camera or RGBD camera. The HUD display image is projected on the windshield, and the monocular camera or RGBD camera is installed behind the windshield to capture the information of the scene in front of the car window to obtain the RGB image. After converting and mapping the RGB image, the brightness distribution of the scene image can be obtained. The controller calculates the target brightness corresponding to each image area according to the calculated brightness distribution of the scene image, and then according to the stored brightness and light source power mapping relationship, The corresponding light source power is obtained, and the light-emitting element in the corresponding image area of the HUD light source is driven with the corresponding light source power value.

Using the vision-based AR-HUD brightness adaptive adjustment method of the present invention, through the method of visual perception, the intensity of the ambient background light is quantified and calculated, and the brightness-light source power mapping relationship is established according to the comfortable brightness range felt by the human eye, and the final adjustment of the HUD is The displayed image brightness, which realizes the dynamic adjustment of the HUD image brightness, can not only improve the overall experience of the driver, but also reduce the overall power of the HUD, alleviating energy consumption and heat dissipation pressure.

The above are only the embodiments of the present invention, and do not limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied to other related technologies In the same way, all fields are included in the scope of patent protection of the present invention.

Claims

A vision-based AR-HUD brightness adaptive adjustment method, which is characterized in that it includes:

Obtain the scene image in front of the car window;

Generating a HUD display image according to the scene image, and acquiring the brightness distribution of the HUD display image;

Dividing the HUD display image into one or more image regions;

Calculating the target brightness corresponding to each of the image regions according to the brightness distribution of the scene image;

The power of the light source corresponding to the image area is adjusted according to the target brightness corresponding to the image area.
The method according to claim 1, wherein the generating a HUD display image according to the scene image comprises:

The scene image in front of the car window is captured, an RGB image and a depth image are obtained according to the scene image, and a HUD display image is generated from the RGB image and the depth image.
The method according to claim 2, wherein the acquiring the brightness distribution of the HUD display image comprises:

The depth image is converted into a 3D point cloud, and the 3D point cloud is mapped to the HUD display image to obtain a brightness distribution.
The method according to claim 3, wherein the dividing the HUD display image into one or more image regions comprises:

Use the entire HUD display image as an image area; or

According to the brightness value in the brightness distribution, the HUD display image is divided into more than one area.
The method according to claim 4, wherein the dividing the HUD display image into more than one area according to the brightness value in the brightness distribution comprises:

An edge detection algorithm is used to extract the edge information of the brightness distribution, and a rectangular frame is used to fit the edge information to obtain several image regions.
The method according to claim 4 or 5, wherein the calculating the target brightness corresponding to each of the image regions according to the brightness distribution of the scene image comprises:

The average brightness value of each image area is calculated, and the target brightness is calculated according to the average brightness value.
The method according to claim 6, wherein the adjusting the light source power corresponding to the image area according to the target brightness corresponding to the image area comprises:

Obtain preset brightness-light source power mapping data, obtain light source power reference values of different image areas according to the target brightness value of each image area, and use the light source power reference values to drive the light-emitting elements of the corresponding image area of the HUD light source.
The method according to claim 3, wherein the conversion formula for converting the depth image into a 3D point cloud is:

Wherein, the z c is the depth value of the depth image pixel, d x and d y are the actual size of the pixel in the photosensitive chip of the camera that shoots the depth image, u o and v o are the center of the camera image plane, f is the focal length of the camera. d x , d y , u o , v o and f are the internal parameters of the camera, and the 3×4 RT matrix is the external parameters of the camera.
The method according to claim 8, wherein the conversion formula for mapping the 3D point cloud onto the image displayed by the HUD is:

Where d x , d y , u o , v o and f are the internal parameters of the camera, the 3×4 RT matrix is the external parameters of the camera, and X w , Y w , and Z w are the 3D points Cloud parameters.
A computer-readable storage medium, wherein the storage medium stores a computer program, and when the program is executed, it is used to implement the steps of the method according to any one of claims 1-9.