WO2022205102A1 - 一种场景处理方法、装置、系统及相关设备 - Google Patents

一种场景处理方法、装置、系统及相关设备 Download PDF

Info

Publication number
WO2022205102A1
WO2022205102A1 PCT/CN2021/084488 CN2021084488W WO2022205102A1 WO 2022205102 A1 WO2022205102 A1 WO 2022205102A1 CN 2021084488 W CN2021084488 W CN 2021084488W WO 2022205102 A1 WO2022205102 A1 WO 2022205102A1
Authority
WO
WIPO (PCT)
Prior art keywords
scene
image
hud
vehicle
outside
Prior art date
Application number
PCT/CN2021/084488
Other languages
English (en)
French (fr)
Inventor
彭惠东
张宇腾
于海
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2021/084488 priority Critical patent/WO2022205102A1/zh
Priority to CN202180001054.8A priority patent/CN113260430B/zh
Publication of WO2022205102A1 publication Critical patent/WO2022205102A1/zh

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/50—Controlling the output signals based on the game progress
    • A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/55—Controlling game characters or game objects based on the game progress
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/80—Special adaptations for executing a specific game genre or game mode
    • A63F13/803—Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/20—Design optimisation, verification or simulation
    • G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/80—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
    • A63F2300/8017—Driving on land or water; Flying
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/80—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
    • A63F2300/8082—Virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2111/00—Details relating to CAD techniques
    • G06F2111/18—Details relating to CAD techniques using virtual or augmented reality

Definitions

  • the embodiments of the present application relate to the field of computer technologies, and in particular, to a scene processing method, apparatus, system, and related equipment.
  • a head up display is a reality device that projects an image into the driver's front view. Compared with the traditional instrument and the central control screen, the driver does not need to lower his head when observing the HUD image, which avoids the back and forth switching of the focal length of the human eye between the image and the road surface, reduces the time for crisis response, and improves driving safety.
  • the augmented reality head up display (AR-HUD) proposed in recent years can further integrate the image projected by the HUD with the real road information to realize augmented reality (AR) navigation and AR early warning. Such functions greatly enhance the driver's access to road information and ensure driving safety and comfort.
  • the embodiments of the present application provide a scene processing method, device, system, and related equipment, which can efficiently and conveniently integrate real information and virtual information to obtain an intuitive AR effect.
  • the scene processing method provided by the embodiment of the present application may be executed by an electronic device or the like.
  • An electronic device refers to a device that can be abstracted into a computer system, wherein an electronic device that supports a scene processing function may also be referred to as a scene processing device.
  • the scene processing device can be the whole machine of the electronic device, such as: a smart phone, a tablet computer, a notebook computer, a desktop computer, a car machine, a car computer or a server, etc.; it can also be an in-vehicle system/device composed of multiple whole machines; It may also be a part of the device in the electronic device, for example: a chip related to scene processing functions, such as a system chip or a scene processing chip, wherein the system chip is also called a system on a chip, or a SoC chip.
  • a chip related to scene processing functions such as a system chip or a scene processing chip, wherein the system chip is also called a system on a chip, or a SoC chip.
  • the scene processing device may be a terminal device such as an in-vehicle machine in an intelligent vehicle, an in-vehicle computer, etc., or a system chip or a scene processing chip that can be installed in a computer system of an intelligent terminal or a surround view system.
  • scenario processing method provided by the embodiment of the present application can be applied to the following scenarios: vehicle simulation system, vehicle game, terminal cloud car viewing, live broadcast, and vehicle field testing and other scenarios.
  • an embodiment of the present application provides a scene processing method, the method includes: acquiring a first outside scene; wherein, the first outside scene is a two-dimensional scene image or a three-dimensional scene model; a first AR image corresponding to the first outside scene; fusing the first outside scene and the first AR image to obtain a second scene; wherein, the first outside scene is the second scene
  • the real information in the scene, the first AR image is the virtual information in the second scene; the display screen is enabled to display the second scene.
  • the embodiments of the present application can construct a large number of two-dimensional or three-dimensional simulation scenes (each simulation scene can including scene elements such as vehicles, roads, or pedestrians); then, based on the above-mentioned simulation scene and the developed model, an AR image corresponding to the above-mentioned simulation scene is generated, and the above-mentioned simulation scene is fused with the corresponding AR image, so as to be fast and efficient.
  • each AR image may include AR icons such as AR navigation guidance arrows and AR warnings.
  • the embodiments of the present application can quickly build a large number of reproducible scenes by means of software without relying on real scenes and HUD hardware devices, with wide coverage and high reusability.
  • the AR function of the HUD can be continuously optimized and improved to ensure the user's use experience.
  • the obtaining a first AR image corresponding to the first outside scene includes: obtaining a first AR image corresponding to the first outside scene and a preset model according to the first outside scene and a preset model.
  • an AR image that matches the simulation scene can be generated by using a pre-trained model, so as to obtain virtual information in augmented reality.
  • the scene includes a crossroad. Based on the navigation information, if the current crossroad should turn right, the corresponding right-turn guidance arrow can be generated.
  • the scene is a cultural scenic spot, the corresponding scenic spot instruction information and scenic spot introduction can be generated. and many more.
  • the preset model is a neural network model
  • the neural network model is based on multiple scenes, multiple AR icons, and differences between the multiple scenes and the multiple AR icons
  • the matching degree is obtained by training.
  • the neural network model can be trained in advance through multiple scenarios, multiple AR icons, and different matching degrees between the multiple scenarios and the multiple AR icons, so that the neural network model can be subsequently trained based on the multiple AR icons. Identify the large number of simulated scenes and generate corresponding AR images, so as to obtain a large number of augmented reality scenes quickly and efficiently.
  • the fusing the first outside scene and the first AR image to obtain the second scene includes: based on a preset head-up display HUD parameter set, in the first A corresponding HUD virtual image surface is determined in the scene outside the vehicle; the HUD virtual image surface is a corresponding area in the first outside scene; the first AR image is rendered into the HUD virtual image surface to obtain a second Scenes.
  • a software-based method can be used to simulate the vehicle-mounted HUD hardware device through an existing computing device, so as to determine a corresponding virtual image plane in the large number of scenes, and the virtual image plane can be an area in the scene.
  • the virtual image surface is the surface projected by the HUD for displaying AR images.
  • an AR image that matches the scene is generated through the pre-trained model, and the AR image is rendered into the virtual image surface.
  • the embodiments of the present application can simulate AR images projected by the HUD in various scenarios by means of software, and complete the AR function (for example, the function includes the above-mentioned preset model for recognizing the scene and generating the corresponding AR image). test, as well. It greatly saves time, resources, manpower and material resources required for real vehicle testing, improves the testing efficiency of AR functions, further ensures the user experience of HUD, and improves the user's driving comfort and safety.
  • the HUD parameter set includes at least one parameter among the curvature of the windshield, the position of the eye box, the observation position of the human eye, the installation position of the HUD, and the size of the HUD virtual image surface.
  • the HUD can be simulated by a software method based on the relevant hardware parameters of the HUD, such as the curvature of the windshield, the position of the eye box, the observation position of the human eye, the installation position of the HUD, and the size of the virtual image surface of the HUD, etc.
  • the corresponding HUD virtual image surface can be constructed in a large number of scenes to provide support for subsequent AR function tests.
  • the embodiments of the present application may not rely on hardware devices, thereby saving manpower and material resources for testers to go out for real vehicle testing, and the method of simulating the HUD through software can more efficiently and cost-effectively implement related AR functions in the HUD, etc. Carry out tests with higher coverage, effectively support the development and testing of AR functions, greatly improve the test efficiency and the reliability of test results, so that a series of software algorithms in the HUD can be better improved and optimized to further ensure users improve the user's driving comfort and safety.
  • the acquiring the first scene outside the vehicle includes: acquiring data collected by a first sensor; the first sensor is a vehicle-mounted sensor; and the data collected by the first sensor is driving in a target vehicle
  • the data collected for the surrounding environment of the target vehicle in the process includes at least one of image data, point cloud data, temperature data and humidity data; the first sensor includes a camera, lidar, millimeter-wave radar, and temperature sensor and at least one of a humidity sensor; and based on the data collected by the first sensor, constructing the first outside scene, where the first outside scene is a real-world simulation scene.
  • the above-mentioned large number of simulation scenarios can be constructed based on a large amount of data collected in computing devices such as computers by means of software (for example, a simulation scene constructed from point cloud data collected by lidar, or a simulation scene constructed by The simulation scene constructed by the image data collected by the camera, or the simulation scene constructed by fusing the data collected by the lidar and the camera, etc., is not specifically limited in this embodiment of the present application).
  • the large number of simulation scenarios are real-world simulation scenarios.
  • the large number of simulation scenarios can be used to support subsequent AR function display and testing, etc., which greatly improves the coverage of the scenario, and ensures the reproducibility of the scenario, thereby ensuring that the improved AR function can be performed in the same way in the future.
  • the test is carried out again in the scene, and the improved AR effect has been verified, etc., which effectively improves the reliability of the test results.
  • the obtaining of the target scene set includes: the obtaining of the first outside scene includes: obtaining data collected by a second sensor; the second sensor is constructed by a preset simulation system The data collected by the second sensor is the data set by the preset simulation system, including at least one of weather, road, pedestrian, vehicle, plant and traffic signal data; based on the second sensor The data collected by sensing is used to construct the first off-vehicle scene, and the first off-vehicle scene is a virtual simulation scene.
  • a plurality of virtual sensors can also be obtained by simulation on a computing device such as a computer by a software method, and the data of each virtual sensor can be set, so that the above-mentioned large number of simulation scenarios can be constructed based on the data of the plurality of virtual sensors, Obviously, the large number of simulation scenes are virtual simulation scenes (similar to building a virtual game scene). In this way, it is possible to further save the time, resources, manpower, and material resources required for the testers to go out to drive the real vehicle to obtain data collected by multiple sensors in the vehicle, and further reduce the test cost.
  • the large number of simulation scenarios can be used to support subsequent AR function display and testing, which greatly improves the coverage of the scenario and ensures the reproducibility of the scenario, thereby ensuring that the improved AR function can be used in the same scenario in the future.
  • the test is carried out again, and the improved AR effect has been verified, etc., which effectively improves the reliability of the test results.
  • the above method further includes: performing a first preprocessing on the first AR image to obtain a second AR image corresponding to the first scene outside the vehicle;
  • the first preprocessing includes at least one of distortion processing and dither processing;
  • the distortion processing includes at least one of radial distortion, tangential distortion, virtual image distance increase and virtual image distance reduction;
  • the dither processing includes superimposing a preset rotational displacement amount and/or the amount of jitter; fusing the first outside scene and the second AR image to obtain a third scene; enabling the display screen to display the third scene.
  • the AR images generated in various scenarios can also be distorted by means of software, for example, including radial distortion, tangential distortion, increased virtual image distance, and decreased virtual image distance.
  • the embodiments of the present application can further perform jitter processing on AR images generated in various scenarios by means of software. For example, different degrees of rotational displacement and jitter can be sequentially superimposed according to actual needs. In this way, the AR imaging effect can be tested and analyzed by extracting a single jitter factor, and intuitively understand the impact of various jitter conditions on the AR imaging effect, so as to provide The subsequent development and testing of the anti-shake function provides effective and accurate support to improve the user experience.
  • the above method further includes: performing a second preprocessing on the second AR image to obtain a third AR image corresponding to the first scene outside the vehicle; the second preprocessing includes at least one of de-distortion processing and anti-shake processing; fusing the first outside scene and the third AR image to obtain a fourth scene; enabling the display screen to display the fourth scene.
  • the above method further includes: based on the third scene and the fourth scene, acquiring the processing effect of the de-distortion processing and/or the anti-shake processing, so as to optimize the corresponding De-distortion function and/or anti-shake function.
  • the de-distortion algorithm to be tested can also be superimposed on the distorted AR image, so that the current de-distortion effect can be intuitively understood, and the de-distortion algorithm can be used for Continue to make corresponding improvements. Then, the AR imaging effect is ensured when the HUD installation position is offset or the curvature of the windshield is inappropriate, so as to ensure the user's visual experience and use experience, and further improve the user's driving comfort and safety.
  • the anti-shake algorithm to be tested can be superimposed on the AR image after the jitter processing, so that the anti-shake effect of the current anti-shake algorithm can be intuitively understood, so as to improve the anti-shake algorithm.
  • the AR imaging effect when the human eye position jitters or the HUD position jitters due to vehicle bumps in actual driving, ensures the user's visual experience and user experience, and further improves the user's driving comfort and safety.
  • the first outside scene includes one or more scene elements; the one or more scene elements include one or more of weather, roads, pedestrians, vehicles, plants and traffic signals each; the one or more AR icons include one or more of left turn, right turn and straight navigation signs; the method further includes: based on the one or more scenarios in each of the second scenarios According to the positional relationship and/or logical relationship between the elements and the one or more AR icons, the preset model is modified accordingly.
  • each simulation scene may include one or more scene elements, such as roads, pedestrians, vehicles, plants, and traffic signals (such as traffic lights, etc.), as well as road signs, overpasses, buildings, and animals, etc. etc., which are not specifically limited in the embodiments of the present application.
  • scene elements such as roads, pedestrians, vehicles, plants, and traffic signals (such as traffic lights, etc.), as well as road signs, overpasses, buildings, and animals, etc. etc., which are not specifically limited in the embodiments of the present application.
  • the AR effect of the preset model is analyzed, and the preset model is corrected accordingly.
  • the simulation scene may include an intersection and some trees beside the road.
  • a corresponding right-turn navigation sign ie, for example, a right-turn guidance arrow
  • the AR The right turn guidance arrow generated in the image does not fit the road well but is displayed on the trees on the left side of the road, or generates an incorrect straight guidance arrow, etc., it can be considered that the current AR effect is not ideal, and the scene elements There is no good match with the positional and/or logical relationship between the corresponding AR icons, and the preset model still needs to be improved.
  • the embodiment of the present application can enable the tester to significantly observe the AR effect of the current preset model on the computing device, or intuitively grasp the AR function of the current HUD by means of simulation, so as to efficiently and accurately Locate the existing problems to better improve and optimize the preset model, so as to ensure the user's use effect and improve the user's driving safety and comfort.
  • an embodiment of the present application provides a scene processing apparatus, including:
  • a first acquiring unit configured to acquire a first scene outside the vehicle; wherein, the first scene outside the vehicle is a two-dimensional scene image or a three-dimensional scene model;
  • a second acquiring unit configured to acquire a first AR image corresponding to the first scene outside the vehicle
  • a fusion unit configured to fuse the first outside scene and the first AR image to obtain a second scene; wherein the first outside scene is the real information in the second scene, and the first scene An AR image is virtual information in the second scene;
  • the first display unit is configured to enable the display screen to display the second scene.
  • the second obtaining unit is specifically used for:
  • the first AR image corresponding to the first outside scene is acquired according to the first outside scene and the preset model; wherein, the first AR image includes one or more AR icons.
  • the preset model is a neural network model
  • the neural network model is based on multiple scenes, multiple AR icons, and differences between the multiple scenes and the multiple AR icons
  • the matching degree is obtained by training.
  • the fusion unit is specifically used for:
  • a corresponding HUD virtual image surface is determined in the first off-vehicle scene; the HUD virtual image surface is a corresponding area in the first off-vehicle scene;
  • the first AR image is rendered into the HUD virtual image surface to obtain a second scene.
  • the HUD parameter set includes at least one parameter among the curvature of the windshield, the position of the eye box, the observation position of the human eye, the installation position of the HUD, and the size of the HUD virtual image surface.
  • the first obtaining unit is specifically used for:
  • Acquire data collected by a first sensor is a vehicle-mounted sensor; the data collected by the first sensor is data collected for the surrounding environment of the target vehicle during the driving process of the target vehicle, including image data, point clouds at least one of data, temperature data, and humidity data; the first sensor includes at least one of a camera, a lidar, a millimeter-wave radar, a temperature sensor, and a humidity sensor;
  • the first outside scene is constructed, and the first outside scene is a real-world simulation scene.
  • the first obtaining unit is specifically used for:
  • the second sensor is a sensor constructed by a preset simulation system
  • the data collected by the second sensor is the data set by the preset simulation system, including weather, Data of at least one of roads, pedestrians, vehicles, plants and traffic signals;
  • the first outside scene is constructed, and the first outside scene is a virtual simulation scene.
  • the apparatus further includes:
  • a first preprocessing unit configured to perform a first preprocessing on the first AR image to obtain a second AR image corresponding to the first outside scene;
  • the first preprocessing includes distortion processing and jitter processing. at least one of;
  • the distortion processing includes at least one of radial distortion, tangential distortion, virtual image distance increase and virtual image distance reduction;
  • the jitter processing includes superimposing a preset rotational displacement amount and/or jitter amount;
  • a second fusion unit configured to fuse the first outside scene and the second AR image to obtain a third scene
  • the second display unit is configured to enable the display screen to display the third scene.
  • the apparatus further includes:
  • a second preprocessing unit configured to perform a second preprocessing on the second AR image to obtain a third AR image corresponding to the first scene outside the vehicle;
  • the second preprocessing includes de-distortion processing and anti-shake processing at least one of the active treatments;
  • a third fusion unit configured to fuse the first outside scene and the third AR image to obtain a fourth scene
  • a third display unit configured to enable the display screen to display the fourth scene.
  • the apparatus further includes:
  • An optimization unit configured to obtain the processing effect of the de-distortion processing and/or the anti-shake processing based on the third scenario and the fourth scenario, so as to optimize the corresponding de-distortion function and/or anti-shake processing Function.
  • the first outside scene includes one or more scene elements; the one or more scene elements include one or more of weather, roads, pedestrians, vehicles, plants and traffic signals each; the one or more AR icons include one or more of the left turn, right turn and straight navigation signs; the device further includes:
  • a correction unit configured to modify the preset model based on the positional relationship and/or logical relationship between the one or more scene elements and the one or more AR icons in each of the second scenes Correction accordingly.
  • an embodiment of the present application provides a scene processing system, including: a terminal and a server;
  • the terminal is configured to send a first off-vehicle scene; wherein, the first off-vehicle scene is sensing information obtained by a sensor of the terminal;
  • the server is configured to receive the first out-of-vehicle scene from the terminal;
  • the server is further configured to acquire a first AR image corresponding to the first outside scene
  • the server is further configured to fuse the first outside scene and the first AR image to obtain a second scene; wherein the first outside scene is the real information in the second scene, and the The first AR image is virtual information in the second scene;
  • the server is further configured to send the second scene
  • the terminal is further configured to receive the second scene and display the second scene.
  • the senor includes at least one of a temperature sensor, a humidity sensor, a global positioning system, a camera, and a lidar; the sensing information includes temperature, humidity, weather, location, image, and point At least one of the clouds.
  • the server is specifically used for:
  • the first AR image corresponding to the first outside scene is acquired according to the first outside scene and the preset model; the first AR image includes one or more AR icons.
  • the preset model is a neural network model
  • the neural network model is based on multiple scenes, multiple AR icons, and differences between the multiple scenes and the multiple AR icons
  • the matching degree is obtained by training.
  • the server is specifically used for:
  • a corresponding HUD virtual image surface is determined in the first off-vehicle scene; the HUD virtual image surface is a corresponding area in the first off-vehicle scene;
  • the first AR image is rendered into the HUD virtual image surface to obtain a second scene.
  • the HUD parameter set includes at least one parameter among the curvature of the windshield, the position of the eye box, the observation position of the human eye, the installation position of the HUD, and the size of the HUD virtual image surface.
  • the server is further used for:
  • the first preprocessing includes at least one of distortion processing and jitter processing;
  • the distortion The processing includes at least one of radial distortion, tangential distortion, virtual image distance increase and virtual image distance reduction;
  • the jitter processing includes superimposing a preset rotational displacement amount and/or a jitter amount;
  • the terminal is further configured to receive the third scene and display the third scene.
  • the server is further used for:
  • the second preprocessing includes at least one of de-distortion processing and anti-shake processing;
  • the fourth scene is sent.
  • the terminal is further configured to receive the fourth scene and display the fourth scene.
  • the server is further used for:
  • the processing effect of the de-distortion processing and/or the anti-shake processing is acquired, so as to optimize the corresponding de-distortion function and/or the anti-shake function.
  • the first outside scene includes one or more scene elements; the one or more scene elements include one or more of weather, roads, pedestrians, vehicles, plants and traffic signals each; the one or more AR icons include one or more of turn left, turn right and straight navigation signs; the server is also used for:
  • the preset model is modified accordingly.
  • an embodiment of the present application provides a computing device, where the computing device includes a processor, and the processor is configured to support the computing device to implement corresponding functions in the scene processing method provided in the first aspect.
  • the computing device may also include a memory for coupling with the processor that holds program instructions and data necessary for the computing device.
  • the computing device may also include a communication interface for the computing device to communicate with other devices or a communication network.
  • the computing device may be a terminal, such as a mobile phone, a car machine, a vehicle-mounted device such as a vehicle-mounted PC, a vehicle such as a car, or a server.
  • the server may be a virtual server or a physical server. It can also be a chip or an electronic system, etc.
  • an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, implements any one of the above-mentioned first aspect.
  • Scene processing method flow may be one or more processors.
  • an embodiment of the present application provides a computer program, where the computer program includes instructions, when the computer program is executed by a computer, the computer can execute the flow of the scene processing method described in any one of the first aspect above.
  • an embodiment of the present application provides a chip system.
  • the chip system may include the scene processing apparatus described in any one of the second aspect above, and is used to implement the scenario described in any one of the first aspect above.
  • the functions involved in the processing method flow.
  • the chip system further includes a memory for storing necessary program instructions and data for the scene processing method.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • an embodiment of the present application provides an electronic device, and the electronic device may include the scene processing device described in any one of the second aspect above, for implementing the scenario described in any one of the first aspect above The functions involved in the processing method flow.
  • the electronic device further includes a memory for storing necessary program instructions and data for the scene processing method.
  • the electronic device may be a terminal, such as a mobile phone, a car machine, a vehicle-mounted device such as a vehicle-mounted PC, a vehicle such as a car, or a server.
  • the server may be a virtual server or a physical server. It can also be a chip or an electronic system, etc.
  • the embodiment of the present application provides a scene processing method, and the embodiment of the present application can build a large number of two-dimensional or three-dimensional simulations by using existing computing devices (such as mobile phones, vehicles, servers, etc.) based on the software simulation method.
  • Scenes each simulation scene can include scene elements such as vehicles, roads, or pedestrians. Then, based on the developed model, an AR image corresponding to the above simulation scene is generated, and the above simulation scene is fused with the corresponding AR image, so as to quickly and efficiently obtain a large number of augmented reality scenes including real information and virtual information.
  • the scene processing method provided by the embodiment of the present application does not need to be equipped with HUD hardware equipment, nor does it need to drive a real car.
  • customers can also simply and directly experience the AR effect of the HUD through the display screen in the store.
  • merchants can upload a large number of augmented reality scenes obtained through simulation to the cloud, and users can use their mobile phones to view the above augmented reality scenes through the corresponding website or application software, so as to understand the functions of the product more quickly and conveniently. Buy products for your own needs.
  • the embodiment of the present application not only saves a lot of manpower and material resources for merchants, but also provides convenience for customers.
  • developers can quickly and intuitively analyze the various models or algorithms involved based on the scene elements and AR icons in each simulation scene that incorporates AR images to optimize corresponding functions.
  • the embodiments of the present application can quickly build various models or a large number of reproducible scenarios required for algorithm testing without relying on real-world scenarios and HUD hardware devices.
  • various models or algorithms can be tested in this large number of scenarios, and AR images projected by HUD in various scenarios can be obtained quickly and intuitively, so as to continuously optimize AR functions and corresponding solutions. Distortion and anti-shake functions, etc., to ensure user experience.
  • the embodiments of the present application can not only greatly improve the test efficiency, scene coverage, and test result reliability of the AR function of the HUD, but also can greatly save the consumption of real vehicle testing. Time resources and manpower and material resources for testers to go out for real vehicle testing, etc., greatly reduce the testing cost.
  • the anchor can collect the information of the surrounding environment (such as weather, roads, vehicles and pedestrians, etc.) in real time through the live broadcast equipment (such as mobile phone terminals, other cameras, etc.), and send the information to the server in real time.
  • the information constructs the simulation scene in real time, restores the real environment around the host, and then generates the corresponding AR image and integrates it with the above-mentioned real-time simulation scene, so as to obtain the real-time augmented reality scene, and send it to the live broadcast device for display by the live broadcast device.
  • the server may not be used, but the live broadcast equipment may be used to directly perform the construction of the simulation scene, the generation and fusion of AR images, and the display of the augmented reality scene, and so on. This is not specifically limited.
  • Figure 1 is a schematic diagram of an AR-HUD imaging effect.
  • FIG. 2 is a functional block diagram of an intelligent vehicle provided by an embodiment of the present application.
  • FIG. 3a is a schematic diagram of a system architecture of a scene processing method provided by an embodiment of the present application.
  • FIG. 3b is a schematic diagram of a system architecture of another scenario processing method provided by an embodiment of the present application.
  • FIG. 3c is a functional block diagram of a computing device provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of an application scenario of a scenario processing method provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of an application scenario of another scenario processing method provided by an embodiment of the present application.
  • FIG. 6a is a schematic flowchart of a scene processing method provided by an embodiment of the present application.
  • FIG. 6b is a schematic flowchart of another scene processing method provided by an embodiment of the present application.
  • FIG. 7 is a schematic flowchart of another scenario processing method provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a scenario provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a camera coordinate system and a world coordinate system provided by an embodiment of the present application.
  • FIG. 10 is a schematic diagram of a scene reconstruction provided by an embodiment of the present application.
  • FIG. 11 is a schematic diagram of a HUD virtual image plane provided by an embodiment of the present application.
  • FIG. 12a is a schematic diagram of an AR imaging effect provided by an embodiment of the present application.
  • FIG. 12b is a schematic diagram of another AR imaging effect provided by an embodiment of the present application.
  • FIG. 12c is a schematic diagram of another AR imaging effect provided by an embodiment of the present application.
  • FIG. 13 is a schematic diagram of a HUD simulation effect from an external perspective provided by an embodiment of the present application.
  • FIG. 14 is a schematic diagram of another AR imaging effect provided by an embodiment of the present application.
  • FIG. 15 is a schematic diagram of a distortion type and AR imaging effect provided by an embodiment of the present application.
  • FIG. 16 is a schematic diagram for comparison of an AR imaging effect provided by an embodiment of the present application.
  • FIG. 17 is a schematic diagram of a human eye line of sight coordinate system and a HUD virtual image plane coordinate system provided by an embodiment of the present application.
  • FIG. 18 is a schematic diagram illustrating the influence of a shaking condition on an AR imaging effect according to an embodiment of the present application.
  • FIG. 19 is a schematic structural diagram of another scene processing apparatus provided by an embodiment of the present application.
  • FIG. 20 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
  • Augmented reality also known as mixed reality. It applies virtual information to the real world through computer technology, and the real environment and virtual objects are superimposed on the same screen or space in real time. Augmented reality provides information that, in general, is different from what humans can perceive. It not only shows the real world information, but also displays the virtual information at the same time, and the two kinds of information complement and superimpose each other.
  • a head-up display also known as a head-up display, is a display device that projects an image into the driver's front view.
  • HUDs were first used in military aircraft to reduce the frequency with which pilots need to look down at instruments. At first, the HUD can project driving-related information on the pilot's helmet through the optical principle, so that the pilot can pay attention to various indicators of flight and receive information transmitted on the ground while ensuring normal driving, thereby improving the safety of driving. and convenience.
  • Today, HUD is being used in automobiles.
  • the driving force of applying HUD is to ensure the safety of driving, so that the driver does not need to lower his head and turn his eyes to the instrument panel or the central control during driving, so as to avoid the focus of the human eye on the instrument. Switching back and forth between the panel or the central control and the road reduces the time for crisis response. Therefore, the information projected by the HUD in the early stage of the design is mainly the driving condition indicators of the car, such as the relatively simple information such as vehicle speed and fuel quantity displayed on the instrument panel.
  • HUDs on the market may include components such as projectors, mirrors (or secondary mirrors), and projection mirrors (or primary mirrors).
  • the imaging principle of the HUD is similar to a slide projection.
  • the driver By projecting the image onto the windshield of the car, the driver can obtain an image of the front of the field of view.
  • light information can be sent out through the projector, reflected on the projection mirror through the reflector, and then reflected on the windshield by the projection mirror.
  • What the human eye sees is a virtual image located about 2-2.5 meters in front of the eyes. The feeling is that information is suspended on the road ahead.
  • the position of the HUD image projected on the windshield is adjustable. Usually, the position of the HUD image can be adjusted by changing the angle of the projection mirror.
  • the windshield of the car is curved, if the image is directly projected on the curved glass surface, the image will be distorted. This requires a corrective action, therefore, projection mirrors and mirrors are often designed to be curved.
  • HUD uses the principle of optical reflection to project and display information such as overspeed warning, vehicle condition monitoring, fuel consumption, and speed on the windshield, allowing the driver to focus on the road ahead and achieve active driving safety. At the same time, it can also reduce the delay and discomfort caused by the need to constantly adjust the focal length of the eyes.
  • AR-HUD augmented reality head up display
  • FIG 1 is a schematic diagram of an AR-HUD imaging effect.
  • the virtual image surface projected by AR-HUD onto the windshield of the car can be located directly in front of the driver's field of vision.
  • AR-HUD can not only project and display basic information such as driving speed and car power, but also display information such as AR navigation guidance arrows, so as to assist drivers to achieve more intelligent , comfortable and safe driving.
  • AR-HUD can realize functions such as AR navigation and AR early warning through the images projected by the HUD.
  • functions such as vehicle following distance warning, line pressure warning, traffic light monitoring, advance lane change indication, pedestrian warning, road sign display, lane departure indication, front obstacle warning, driver status monitoring, etc. Let's go into details.
  • the embodiments of the present application provide a series of solutions, which are based on existing computing devices (such as mobile phones, vehicles, servers, etc.), through software
  • various scenarios can be constructed, so as to simulate AR functions in these scenarios, including generating AR images corresponding to the above scenarios, and fusing various scenarios with the corresponding AR images, so as to quickly and efficiently obtain A large number of augmented reality scenes including real information and virtual information, so as to experience the AR effect in each scene conveniently and intuitively.
  • the series of solutions provided in the embodiments of the present application may also test AR functions according to the simulated augmented reality scenarios, so as to continuously improve and optimize the AR functions and enhance the AR effects.
  • FIG. 2 is a functional block diagram of an intelligent vehicle provided by an embodiment of the present application.
  • a scenario processing method provided by this embodiment of the present application may be applied to the smart vehicle 200 as shown in FIG. 2 .
  • the smart vehicle 200 may be configured in a fully or partially automatic driving mode.
  • the intelligent vehicle 200 may be set to operate without human interaction.
  • Intelligent vehicle 200 may include various subsystems, such as travel system 202 , sensing system 204 , control system 206 , one or more peripherals 208 and power supply 210 , computer system 212 , and user interface 216 .
  • intelligent vehicle 200 may include more or fewer subsystems, and each subsystem may include multiple elements. Additionally, each of the subsystems and elements of the intelligent vehicle 200 may be wired or wirelessly interconnected.
  • the travel system 202 may include components that provide powered motion for the intelligent vehicle 200 .
  • travel system 202 may include engine 218 , energy source 219 , transmission 220 , and wheels 221 .
  • Engine 218 may be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a gasoline engine and electric motor hybrid engine, an internal combustion engine and an air compression engine hybrid engine.
  • Engine 218 may convert energy source 219 into mechanical energy.
  • Examples of energy sources 219 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 219 may also provide energy to other systems of intelligent vehicle 200 .
  • Transmission 220 may transmit mechanical power from engine 218 to wheels 221 .
  • Transmission 220 may include a gearbox, a differential, and a driveshaft.
  • transmission 220 may also include other devices, such as clutches.
  • the drive shafts may include one or more axles that may be coupled to one or more wheels 221 .
  • the sensing system 204 may include several sensors that may be used to collect environmental information about the surroundings of the intelligent vehicle 200 (eg, may include terrain, roads, motor vehicles, non-motor vehicles, pedestrians, roadblocks, traffic signs, traffic lights, animals, buildings and plants, etc.). As shown in FIG. 2, the sensing system 204 may include a positioning system 222 (the positioning system may be a global positioning system (GPS) system, a Beidou system or other positioning systems), an inertial measurement unit (inertial measurement unit) , IMU) 224, radar 226, laser rangefinder 228, camera 230, and computer vision system 232, among others.
  • GPS global positioning system
  • IMU inertial measurement unit
  • the tester can drive the intelligent vehicle 200 to various driving environments (for example, driving environments in different regions, different terrains, different road conditions and different weather), and pass the sensor system 204.
  • the multiple sensors of the device collect data of the surrounding environment, and further, the collected data can be uploaded to the server.
  • testers can obtain a large amount of sensor data collected in various environments from the server, and build a large number of corresponding scenarios through computing devices based on the large amount of sensor data.
  • the tester may also directly send a large amount of collected data to a computing device, etc., which is not specifically limited in this embodiment of the present application.
  • the smart vehicle 200 may also include an AR-HUD (that is, a HUD with AR functions, not shown in FIG. 2 ), which can project AR images to the front of the driver's field of vision, so as to facilitate the operation during driving. Safe and comfortable AR navigation and AR early warning, etc., the AR-HUD can be the AR-HUD that has been tested and improved by the above simulation.
  • an AR-HUD that is, a HUD with AR functions, not shown in FIG. 2
  • the AR-HUD can be the AR-HUD that has been tested and improved by the above simulation.
  • the above-mentioned server may be a server, a server cluster composed of multiple servers, or a cloud computing service center, etc., which are not specifically limited in this embodiment of the present application.
  • the above computing device may be a smart wearable device, a smart phone, a tablet computer, a notebook computer, a desktop computer, or a server with a display screen, etc., which are not specifically limited in this embodiment of the present application.
  • the positioning system 222 may be used to estimate the geographic location of the intelligent vehicle 200 .
  • the IMU 224 is used to sense position and orientation changes of the intelligent vehicle 200 based on inertial acceleration.
  • IMU 224 may be a combination of an accelerometer and a gyroscope.
  • Radar 226 may utilize radio signals to sense objects within the surrounding environment of intelligent vehicle 200 .
  • the radar 226 may also be used to sense the speed and/or direction of travel of vehicles surrounding the smart vehicle 200 , among others.
  • the radar 226 can be a lidar, a millimeter-wave radar, etc., and can be used to collect point cloud data of the surrounding environment, and then a large number of scenes in the form of point clouds can be obtained for simulation testing of the HUD.
  • the laser rangefinder 228 may utilize laser light to sense objects in the environment in which the intelligent vehicle 200 is located.
  • laser rangefinder 228 may include one or more laser sources, one or more laser scanners, and one or more detectors, among other system components.
  • the camera 230 can be used to capture multiple images of the surrounding environment of the smart vehicle 200, and then a large number of scenes in the form of photos can be obtained for simulation testing of the HUD.
  • camera 230 may be a still camera or a video camera.
  • Computer vision system 232 is operable to process and analyze images captured by camera 230 in order to identify objects and/or features in the environment surrounding intelligent vehicle 200 .
  • the objects and/or features may include terrain, motor vehicles, non-motor vehicles, pedestrians, buildings, traffic signals, road boundaries and obstacles, and the like.
  • Computer vision system 232 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques.
  • SFM structure from motion
  • the control system 206 controls the operation of the intelligent vehicle 200 and its components.
  • Control system 206 may include various elements, including throttle 234 , braking unit 236 , and steering system 240 .
  • the throttle 234 is used to control the operating speed of the engine 218 and thus the speed of the intelligent vehicle 200 .
  • the braking unit 236 is used to control the deceleration of the intelligent vehicle 200 .
  • the braking unit 236 may use friction to slow the wheels 221 .
  • the braking unit 236 may convert the kinetic energy of the wheels 221 into electrical current.
  • the braking unit 236 may also take other forms to slow down the wheels 221 to control the speed of the smart vehicle 200 .
  • Steering system 240 is operable to adjust the heading of intelligent vehicle 200 .
  • control system 206 may additionally or alternatively include components other than those shown and described. Alternatively, some of the components shown above may be reduced.
  • the intelligent vehicle 200 interacts with external sensors, other vehicles, other computer systems, or the user through peripheral devices 208.
  • Peripherals 208 may include a wireless communication system 246 , an onboard computer 248 , a microphone 250 and/or a speaker 252 .
  • the data collected by one or more sensors in the sensor system 204 may also be uploaded to the server through the wireless communication system 246 , and the data collected by one or more sensors in the sensor system 204 may also be uploaded through the wireless communication system 246
  • the data collected by the sensor is sent to a computing device for performing a simulation test on the HUD, etc., which is not specifically limited in this embodiment of the present application.
  • peripherals 208 provide a means for a user of intelligent vehicle 200 to interact with user interface 216 .
  • the onboard computer 248 may provide information to the user of the smart vehicle 200 .
  • User interface 216 may also operate on-board computer 248 to receive user input.
  • the onboard computer 248 can be operated via a touch screen.
  • peripheral devices 208 may provide a means for intelligent vehicle 200 to communicate with other devices located within the vehicle.
  • microphone 250 may receive audio (eg, voice commands or other audio input) from a user of intelligent vehicle 200 .
  • speaker 252 may output audio to a user of intelligent vehicle 200 .
  • Wireless communication system 246 may wirelessly communicate with one or more devices, either directly or via a communication network.
  • wireless communication system 246 may use 3rd generation mobile networks (3G) cellular communications, such as code division multiple access (CDMA), global system for mobile communications, GSM)/General Packet Radio Service (GPRS), or 4th Generation Mobile Networks (4G) cellular communications, such as Long Term Evolution (LTE). Or 5th generation mobile networks (5G) cellular communications.
  • the wireless communication system 246 may also utilize wireless-fidelity (WIFI) to communicate with a wireless local area network (WLAN).
  • WIFI wireless-fidelity
  • WLAN wireless local area network
  • the wireless communication system 246 may communicate directly with the device using an infrared link, Bluetooth, or the like.
  • Other wireless protocols, such as various vehicle communication systems, for example, wireless communication system 246 may include one or more dedicated short range communications (DSRC) devices, which may include a combination of vehicle and/or roadside stations. public and/or private data communications between them.
  • DSRC dedicated short
  • Power supply 210 may provide power to various components of intelligent vehicle 200 .
  • the power source 210 may be a rechargeable lithium-ion or lead-acid battery.
  • One or more battery packs of such batteries may be configured as a power source to provide power to various components of the intelligent vehicle 200 .
  • power source 210 and energy source 219 may be implemented together, such as in some all-electric vehicles.
  • Computer system 212 may include at least one processor 213 that executes instructions 215 stored in a non-transitory computer-readable medium such as memory 214 .
  • Computer system 212 may also be multiple computing devices that control individual components or subsystems of intelligent vehicle 200 in a distributed fashion.
  • the processor 213 may be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, the processor may be a dedicated device such as an application-specific integrated circuit (ASIC) or other hardware-based processor.
  • FIG. 2 functionally illustrates the processor, memory, and other elements of the computer system 212 in the same block, one of ordinary skill in the art will understand that the processor or memory may actually include a processor or memory that is not stored in the same physical enclosure multiple processors or memories within.
  • the memory may be a hard drive or other storage medium located within an enclosure other than computer system 212 .
  • a reference to a processor or memory will be understood to include a reference to a collection of processors or memories that may or may not operate in parallel.
  • some of the components in sensing system 204 may each have its own processor that only performs computations related to component-specific functions .
  • the processor 213 may be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle while others are performed by a remote processor.
  • memory 214 may include instructions 215 (eg, program logic) executable by processor 213 to perform various functions of intelligent vehicle 200, including those described above.
  • Memory 214 may also contain additional instructions, including sending data to, receiving data from, interacting with and/or controlling one or more of travel system 202 , sensing system 204 , control system 206 and peripherals 208 instruction.
  • the memory 214 can also store data, such as a large amount of sensor data collected by the sensor system 204 during driving, such as image data captured by the camera 230 within the sensor system 204 and point clouds collected by the radar 226 . data, etc.
  • the memory 214 may also store, for example, road maps, route information, the vehicle's position, direction, speed, and other such vehicle data, as well as other information, among others. Such information may be used by the wireless communication system 246 or the computer system 212 or the like in the intelligent vehicle 200 during travel of the intelligent vehicle 200 .
  • User interface 216 for providing information to or receiving information from a user of intelligent vehicle 200 .
  • the user interface 216 may include one or more input/output devices within the set of peripheral devices 208 , such as a wireless communication system 246 , an onboard computer 248 , a microphone 250 and a speaker 252 .
  • one or more of these components described above may be installed or associated with the intelligent vehicle 200 separately.
  • memory 214 may exist partially or completely separate from intelligent vehicle 200 .
  • the above-described components may be communicatively coupled together in a wired and/or wireless manner.
  • the smart vehicle 200 can be a car, a truck, a motorcycle, a bus, a boat, a drone, an airplane, a helicopter, a lawn mower, a recreational vehicle, a playground vehicle, construction equipment, a tram, a golf ball A car, a train, and a trolley, etc., which are not specifically limited in this embodiment of the present application.
  • the functional block diagram of the smart vehicle in FIG. 2 is only an exemplary implementation in the embodiment of the present application, and the smart vehicle in the embodiment of the present application includes but is not limited to the above structures.
  • FIG. 3a is a schematic diagram of a system architecture of a scene processing method provided by an embodiment of the present application.
  • the technical solution of the embodiment of the present application may be implemented in the system architecture illustrated in FIG. 3a or a similar system architecture.
  • the system architecture may include the computing device 100 and a plurality of smart vehicles, and may specifically include smart vehicles 200a, 200b and 200c.
  • the computing device 100 and the smart vehicles 200a, 200b, and 200c may establish connections with each other through wired or wireless networks (eg, Wireless-Fidelity (WiFi), Bluetooth, and mobile networks).
  • WiFi Wireless-Fidelity
  • Bluetooth Wireless-Fidelity
  • the smart vehicles 200a, 200b, and 200c may be installed with multiple sensors, such as cameras, lidars, millimeter-wave radars, temperature sensors, humidity sensors, and so on.
  • the smart vehicles 200a, 200b and 200c can collect data of the surrounding environment (which may include data such as images, point clouds, temperature and humidity, etc.) through multiple sensors in the vehicle, and send the data in a wired or wireless manner. to computing device 100 .
  • the computing device 100 can construct a large number of simulation scenes based on the large amount of sensor data, generate AR images corresponding to each scene, and fuse with them to obtain a corresponding augmented reality scene, and finally display the obtained augmented reality scene.
  • FIG. 3b is a schematic diagram of the system architecture of another scenario processing method provided by the embodiment of the present application.
  • the technical solution of the embodiment of the present application may be the system architecture shown in the example of FIG. 3b or a similar system. specific implementation in the architecture.
  • the system architecture may further include a server 300 in addition to the aforementioned computing device 100 and the smart vehicles 200a, 200b and 200c.
  • the data collected by the smart vehicles 200a, 200b and 200c during the driving process can be uploaded to the server 300 through the network, and the subsequent computing device 100 can obtain the corresponding sensor data from the server through the network, and so on. It will not be repeated here.
  • the sharing of sensor data can be realized more conveniently, so that different testers can obtain the data collected by the smart vehicles 200a, 200b and 200c from the server through the network according to actual needs, thereby improving the test efficiency.
  • the computing device 100 can construct a large number of simulation scenes based on the large amount of sensor data, generate AR images corresponding to each scene, and associate them with the AR images. Fusion is performed to obtain corresponding augmented reality scenes (that is, AR effects in various scenes), and finally the obtained augmented reality scenes are displayed. So far, the computing device 100 has completed the method of software simulation, which does not depend on the HUD hardware device, and efficiently and intuitively obtains AR effects in various scenarios, which greatly reduces the cost of AR function display. The AR effects in various scenarios will comprehensively and efficiently test the AR functions of the HUD, so as to continuously optimize the AR functions and ensure the user experience.
  • the tester can drive the smart vehicles 200a, 200b and 200c to various terrains, road conditions, weather and other driving environments to collect data of various scenarios, so as to simulate and test the AR function of the subsequent HUD, as well as jitter or
  • the AR imaging effect simulation test under the influence of distortion factors provides support for a large number of reproducible scenes, which greatly improves the reliability of the test results.
  • the server 300 receives the sensor data uploaded by the smart vehicles 200a, 200b and 200c, and the data may be the original data collected by the sensor, or the data after sensor preprocessing (such as screening, fusion, etc.) ,and many more.
  • the server 300 may construct a large number of simulation scenes based on the large amount of sensor data by using the method in this embodiment of the present application, generate AR images corresponding to each scene, and fuse them to obtain a corresponding augmented reality scene.
  • the server 300 sends the obtained augmented display scene to the computing device 100 through a message, so that the computing device 100 displays the augmented reality scene through the display screen.
  • the computing device 100 may also be a display screen or the like in the smart vehicles 200a, 200b and 200c.
  • the data collected by the on-board sensors may also include the data collected by the live broadcast equipment (not shown in the figure) on the vehicle, For example, video data of the surrounding environment collected by the host's mobile phone terminal, etc.
  • the server 300 can receive the data collected in real time, and based on this, build a simulation scene in real time, restore the environment around the host, and generate and fuse AR images corresponding to it in real time to obtain a corresponding augmented reality scene, and The obtained augmented reality scene is sent to the live broadcast device, so that the display screen of the live broadcast device displays the augmented reality scene.
  • the audience can use the mobile phone or tablet in their hand to watch the real-time augmented reality scene (for example, including the tourist scene shot by the anchor and the AR icons such as weather, scenic spot logo and scenic spot introduction, etc. integrated by the server) through the online live broadcast room.
  • the scene data used by the server 300 to perform the simulation may be data collected in advance or real-time data, which is not specifically limited in this embodiment of the present application.
  • FIG. 3c is a functional block diagram of a computing device provided by an embodiment of the present application.
  • the computing device 100 may include a human eye posture simulation module 101, a HUD parameter simulation module 102, a sensor data module 103, a data fusion module 104, an AR function module 105, a rendering engine module 106, a HUD virtual module 107, Scene generation module 108 and display module 109 .
  • the human eye posture simulation module 101 can be used to simulate the human eye posture of the driver during driving, including the shaking change of the driver's eye position when the vehicle is bumpy. It can be understood that the change of the human eye posture can affect the The AR imaging effect projected by the HUD as observed by the driver.
  • the HUD parameter simulation module 102 can be used to set HUD-related hardware parameters. It should be understood that, as mentioned above, the HUD's own hardware or different curvatures of the windshield are likely to cause distortion of the AR image. Therefore, various hardware parameters can be parameterized and simulated by the HUD parameter simulation module 102, thereby providing a large number of, Comprehensive input parameters. For example, parameters such as the curvature of the windshield, the position of the eye box, the size of the HUD virtual image surface, and the HUD installation position may be included, which are not specifically limited in this embodiment of the present application.
  • the sensor data module 103 can be used to acquire and store a large amount of data collected by multiple sensors (for example, the data collected by multiple sensors in the above-mentioned smart vehicles 200a, 200b, and 200c during driving).
  • the sensor data may also be data set in a virtual sensor constructed by an existing simulation simulation system (or simulation software), and so on.
  • the foregoing system architecture may also only include computing devices, which are not specifically limited in this embodiment of the present application. Therefore, there is no need for testers to assemble sensors in the vehicle and drive the vehicle to the actual road for data collection, which further saves manpower, material resources and time resources, greatly improves the test efficiency and reduces the test cost, and the virtual sensor does not depend on the actual road.
  • Cars and real scenes through simple data settings and changes, can build a variety of different scenes more comprehensively and efficiently. It can be understood that the tester can select real vehicle to collect data or set virtual sensor data according to actual conditions and test requirements, etc., which is not specifically limited in this embodiment of the present application.
  • the data fusion module 104 can be used to fuse the data in the sensor module 103, such as the image data captured by the camera and the point cloud data collected by the lidar, etc.
  • the scene provides more comprehensive and effective data support to improve the quality of the scene.
  • the data fusion module 104 can also receive the relevant data in the human eye posture simulation module 101 and the HUD parameter simulation module 102, and then can intuitively display it in the virtual image plane projected by the subsequent virtual HUD device.
  • the AR function module 105 may include a corresponding series of models and software algorithms, etc., and may be used to generate AR images that match various scenes, and each AR image may include one or more AR icons, such as AR navigation guide arrows, driving speed and car battery, etc.
  • the rendering engine module 106 may be used to render the corresponding one or more AR icons into the virtual image surface projected by the HUD.
  • the HUD virtual module 107 can be used to simulate the HUD based on the relevant parameters in the above-mentioned HUD parameter simulation module 102, and construct a corresponding HUD virtual image surface, and so on.
  • the scene generation module 108 can be used to directly construct a large number of scenes based on the sensor data in the above-mentioned sensor data module 103, and can also be used to construct a large number of scenes based on the data obtained by fusing various types of sensor data in the above-mentioned data fusion module 104. ,and many more. Further, corresponding HUD virtual image surfaces can also be constructed in the large number of scenes through the above-mentioned HUD virtual module 107 .
  • the display module 109 can be used to display the AR image generated based on the current scene, the model and the aforementioned corresponding hardware parameters and human eye posture in the HUD virtual image plane of the current scene, so that the tester can intuitively grasp the AR image in the current situation.
  • the imaging effect and further analyze the possible problems of the model or algorithm according to the AR imaging effect, so as to optimize the AR function. And it can also analyze the impact of various hardware parameters and changes in human eye posture on the AR imaging effect according to the AR imaging effect, so as to provide effective support for the subsequent development and testing of the anti-distortion function and anti-shake function, and continuously improve the use of users. experience.
  • FIG. 3c the functional block diagram of the computing device in FIG. 3c is only an exemplary implementation in the embodiments of the present application, and the computing devices in the embodiments of the present application include but are not limited to the above structures.
  • the computing device 100 may be a smart phone, a smart wearable device, a tablet computer, a notebook computer, a desktop computer, a car machine, etc. with the above functions, which are not specifically limited in this embodiment of the present application.
  • the smart vehicles 200a, 200b, and 200c may be domestic cars, vans, buses, taxis, motorcycles, yachts, etc. with the above functions, which are not specifically limited in this embodiment of the present application.
  • the server 300 can be a computer and a server with the above functions, etc.
  • the server 300 can be a server, a server cluster composed of multiple servers, or a cloud computing service center, and the server 300 can be a computing server.
  • the device 100 and the smart vehicles 200a, 200b, and 200c provide background services, such as a car networking service platform, etc., which are not specifically limited in this embodiment of the present application.
  • Scenario 1 based on the data collected by the vehicle sensors for the real scene, build a scene, and conduct a simulation test on the HUD.
  • FIG. 4 is a schematic diagram of an application scenario of a scenario processing method provided by an embodiment of the present application.
  • the application scenario may include the computing device 100 and a smart vehicle 200 (taking a car as an example in FIG. 4 ) driving on an actual road (for example, a multi-lane highway that is more common in the real world as shown in FIG. 4 ).
  • the application scenario may also include multiple other vehicles, such as vehicle 1 (taking a bus as an example in FIG. 4 ), vehicle 2 (taking a car as an example in FIG. 4 ), and a vehicle. 3 (take a car as an example in Figure 4).
  • the communication connection between the computing device 100 and the smart vehicle 200 can be established in a wired or wireless manner, and the smart vehicle 200 can be any one of the above-mentioned smart vehicles 200a, 200b, and 200c, with a built-in sensor system, including multiple sensors (such as cameras, Lidar and mmWave radar, etc.).
  • the intelligent vehicle 200 is driving on the road, the data of the surrounding environment can be collected by multiple sensors of the vehicle, and the data can be sent to the computer device 100 .
  • the computing device 100 can construct a large number of scenarios based on data collected by multiple sensors in the smart vehicle 200 by using a scenario processing method provided in an embodiment of the present application.
  • the computing device 100 can construct a virtual HUD based on preset HUD hardware parameters by means of software simulation, so as to generate and fuse corresponding AR images in the multiple real-life simulation scenarios, so as to obtain a large number of AR images efficiently and conveniently.
  • Augmented reality scene and display in this way, users can intuitively experience the AR functions of HUD products in various scenarios without driving the real car to the road, which provides convenience for users.
  • the AR function of the HUD can also be tested based on the large number of augmented reality scenarios.
  • the embodiment of the present application can comprehensively and comprehensively test the AR function of the HUD in various scenarios without relying on the HUD hardware device. Efficient testing to continuously optimize AR functions, ensure user experience, and greatly reduce testing costs.
  • the computing device 100 may be a smart phone, a smart wearable device, a tablet computer, a notebook computer, a desktop computer, a car computer, etc. with the above functions, which are not specifically limited in this embodiment of the present application.
  • the smart vehicle 200 may be a family car, a van, a bus, a taxi, a motorcycle, a yacht, etc. with the above functions, which is not specifically limited in the embodiment of the present application.
  • Scenario 2 build a scenario based on the data set in the virtual sensor, and conduct a simulation test on the HUD.
  • FIG. 5 is a schematic diagram of an application scenario of another scenario processing method provided by an embodiment of the present application.
  • the application scenario may include the computing device 100 .
  • the computing device 100 may first construct a plurality of virtual sensors (for example, may include virtual cameras, virtual lidars, virtual millimeter-wave radars, etc.) through an existing simulation simulation system (or simulation software), and then The data of each virtual sensor is set, such as setting virtual objects in the virtual camera, such as virtual vehicles, virtual roads and so on. Then, the computing device 100 may construct multiple scenarios based on the data of the multiple virtual sensors, and obviously, correspondingly, the multiple scenarios may all be virtual simulation scenarios. As shown in FIG.
  • the virtual simulation scene may be similar to a game scene from a first-person perspective, etc., which is not specifically limited in this embodiment of the present application.
  • the computing device 100 can construct a virtual HUD based on preset HUD hardware parameters by means of software simulation, so as to generate and fuse corresponding AR images in the multiple real-life simulation scenarios, so as to obtain a large number of AR images efficiently and conveniently. Augmenting the reality scene and displaying it, etc., will not be repeated here.
  • the application scenario shown in FIG. 5 further saves the time, resources, manpower and material resources that the staff needs to go out to collect data. , which further reduces the cost of AR function display and AR function test, and improves the test efficiency.
  • the computing device 100 may be a smart phone, a smart wearable device, a tablet computer, a notebook computer, a desktop computer, a car computer, etc. with the above functions, which are not specifically limited in this embodiment of the present application.
  • Scenario 3 Live broadcast based on real-time collected data.
  • the data collected by the on-board sensors (which may also include live broadcast equipment) is collected by the host while driving the vehicle 200.
  • the collected data (not shown in the figure, such as video data of the surrounding environment collected by the host's mobile terminal, etc.) can be uploaded to the computing device 100 in real time.
  • the computing device 100 can receive the data collected in real time, and based on this, construct a simulation scene in real time, restore the environment around the host, and generate and fuse AR images corresponding to it in real time to obtain a corresponding augmented reality scene, and
  • the obtained augmented reality scene is sent to the live broadcast device, so that the display screen of the live broadcast device displays the augmented reality scene.
  • the audience can use the mobile phone or tablet in their hand to watch the real-time augmented reality scene (for example, including the tourist scene shot by the anchor and the AR icons such as weather, scenic spot logo and scenic spot introduction, etc. integrated by the server) through the online live broadcast room.
  • the computing device 100 may be a server with the above functions, may be a server, may be a server cluster composed of multiple servers, or a cloud computing service center, etc., which is not specifically limited in this embodiment of the present application .
  • FIG. 6a is a schematic flowchart of a scene processing method provided by an embodiment of the present application.
  • the method can be applied to the system architecture described in Fig. 3a and Fig. 3b and the application scenarios described above in Fig. 4 and Fig. 5, where the computing device can be used to support and execute the method flow steps S801- Step S804.
  • the following will be described from the computing device side with reference to FIG. 6a, the method may include the following steps S801-S804.
  • Step S801 acquiring a first outside scene; wherein, the first outside scene is a two-dimensional scene image or a three-dimensional scene model.
  • the first outside scene of the computing device may be a two-dimensional scene image, or a three-dimensional scene model, etc., which is not specifically limited in this embodiment of the present application.
  • the scene elements may include, for example, lanes, vehicles, plants, pedestrians, animals, and traffic signals, etc., which are not specifically limited in this embodiment of the present application.
  • Step S802 acquiring a first AR image corresponding to the first scene outside the vehicle.
  • the computing device may identify the first outside scene based on the first outside scene and the preset model, and obtain a first AR image corresponding to the first outside scene.
  • each first AR image may include one or more AR icons (for example, warning icons or text information for paying attention to vehicles ahead and sidewalks, etc., and for example, related attraction introductions and weather information, etc., and may also include related AR navigation signs, such as straight arrows and left/right turn arrows, etc., which are not specifically limited in this embodiment of the present application).
  • each first out-of-vehicle scene and each AR icon may have corresponding one or more attributes, such as weather, season, geographic location, road conditions, terrain, traffic signals, and the like.
  • the attribute of each AR icon included in the first AR image may be the same as (or match) the attribute of the first scene outside the vehicle. For example, if the attributes of the first off-vehicle scene include school and road driving, then based on the first off-vehicle scene and the preset model, the corresponding school logo and AR icons such as slowing down and slowing down may be acquired.
  • Step S803 fusing the first outside scene and the first AR image to obtain a second scene.
  • the computing device may fuse the first outside scene and the corresponding first AR image to obtain the corresponding second scene.
  • the first outside scene is real information in the second scene
  • the first AR image is virtual information in the second scene.
  • Step S804 enabling the display screen to display the second scene.
  • the display screen may be a display screen of the computing device, and the computing device may display the second scene through its display after obtaining the second scene.
  • the display screen may also be a display screen of another device, and after obtaining the second scene, the computing device may send the second scene to other devices, so that the other device displays the first scene through its display screen.
  • the embodiments of the present application can construct a large number of two-dimensional or three-dimensional simulation scenarios by using existing computing devices (such as notebook computers and desktop computers, etc.) based on software methods; then, based on the above-mentioned simulation scenarios and developed models, generate The AR image corresponding to the above simulation scene, and the above simulation scene is fused with the corresponding AR image, so as to obtain a large number of augmented reality scenes including real information and virtual information quickly and efficiently.
  • existing computing devices such as notebook computers and desktop computers, etc.
  • the embodiments of the present application may not rely on real scenes and HUD hardware devices, and not only can display various augmented reality scenarios quickly and efficiently, but also can follow various augmented reality simulations obtained by the embodiments of the present application. Intuitive AR effects in the scene, and constantly optimize and improve the AR function of the HUD to ensure the user experience.
  • FIG. 6b is a schematic flowchart of another scene processing method provided by an embodiment of the present application.
  • the method can be applied to the system architecture described in Fig. 3a and Fig. 3b and the application scenarios described above in Fig. 4 and Fig. 5, where the computing device can be used to support and execute the method flow steps S901- Step S903.
  • the method may include the following steps S901-S903.
  • Step S901 Obtain a target scene set, where the target scene set includes X first outside scenes; each first outside scene includes one or more scene elements.
  • the computing device acquires a target scene set, where the target scene set may include X first outside scenes, and each first outside scene may include one or more scene elements.
  • the target scene set may include X first outside scenes, and each first outside scene may include one or more scene elements.
  • X is an integer greater than or equal to 1.
  • the computing device may acquire data collected by N first sensors, and the N first sensors may be on-board sensors (for example, any one of the above-mentioned smart vehicles 200a, 200b, and 200c) disposed on the target vehicle. such as cameras, lidars, and millimeter-wave radars).
  • the data collected by the N first sensors may be data collected for the surrounding environment of the target vehicle during the driving of the target vehicle.
  • N is an integer greater than or equal to 1.
  • the computing device may use an existing rendering engine (eg, Open Graphics Library (openGL), unreal (Unreal Engine), and Unity (a rendering engine) etc.) to construct the corresponding K first outside scenes.
  • an existing rendering engine eg, Open Graphics Library (openGL), unreal (Unreal Engine), and Unity (a rendering engine) etc.
  • the K first outside scenes may be real-life simulation scenes.
  • K is an integer greater than or equal to 1 and less than or equal to X.
  • the computing device may further acquire data collected by M second sensors, where the M second sensors may be virtual sensors constructed by a preset simulation system.
  • the data collected by the M second sensors may be data set by the above-mentioned preset simulation system.
  • M is an integer greater than or equal to 1.
  • the computing device may construct corresponding P first exterior scenes by using an existing rendering engine based on the data collected by the M second sensors.
  • the P first outside scenes may be virtual simulation scenes (similar to virtual driving game scenes).
  • P is an integer greater than or equal to 1 and less than or equal to X.
  • Step S801 may refer to the method flow shown in FIG. 7 .
  • the computing device can use the existing simulation system to initialize the basic scene settings, for example, it can include basic unit information such as traffic, map, weather, vehicle, and various sensors.
  • the tester can choose to generate the corresponding real-world simulation scene (such as the above K first scene outside the car) through the sensor data collected by the real car driving, or can choose to set the virtual scene
  • the data in the sensor generates a corresponding virtual simulation scene (for example, the above-mentioned P first outside scenes).
  • step S12a and step S13a in FIG. 7 if you choose not to pass the real vehicle data, the rendering engine will complete the information of each basic unit through the data input of the virtual sensor in a manner similar to the construction of the virtual driving game scene.
  • the instantiation operation of that is, the entire scene will be generated by means of simulation.
  • FIG. 8 is a schematic diagram of a scenario provided by an embodiment of the present application.
  • the first scene outside the vehicle is a virtual simulation scene (a two-dimensional scene image is taken as an example in FIG.
  • the first outside scene may also include elements such as a steering wheel, a car windshield, and a car hood from a first-person perspective of the driver (not shown in FIG. 8 ), which is not made in this embodiment of the present application.
  • a 3D stereoscopic scene model can also be obtained based on the data input of the virtual sensor and constructed through an existing rendering engine, and then the HUD can be subsequently performed in the 3D stereoscopic scene model. AR function simulation test, so as to improve the observability and accuracy of the test, etc., which is not specifically limited in this embodiment of the present application.
  • the computing device can read data such as images and point clouds in the corresponding sensors through the rendering engine, and use SFM/synchronized positioning and map construction (Simultaneous Localization And Mapping, SLAM) algorithm is used for offline/online 3D sparse/dense scene reconstruction, and at the same time, the basic unit is instantiated with the data of sensors such as lidar, GPS/IMU, and the whole scene will be generated by real scene synthesis.
  • the generated scene can be imported into the rendering engine through storage forms such as object (object file) and mesh (mesh).
  • the scene generated by the data collected by the real sensor may be the above-mentioned scene, or may be a three-dimensional scene model, etc., which will not be repeated here.
  • FIG. 9 is a schematic diagram of a camera coordinate system and a world coordinate system provided by an embodiment of the present application.
  • the relationship between the three-dimensional (3 dimensions, 3D) coordinates ⁇ x, y, z ⁇ of objects in the scene and the 2D coordinates ⁇ u, v ⁇ of the image captured by the real camera sensor should be satisfied:
  • f x and f y are the focal lengths
  • c x and cy are the offsets of O o from the center O c
  • K is the camera internal parameter.
  • FIG. 10 is a schematic diagram of scene reconstruction provided by an embodiment of the present application.
  • local scene reconstruction can be performed through a single camera image or point cloud data of a single scene to obtain a single-frame scene.
  • Figure 10 Through the reconstruction technology of multi-frame images, the multi-modal data collected by the sensor (such as multiple camera images from different angles and different scenes, and multiple lidar images collected for different scenes can be collected).
  • Point cloud data, etc. are fused to reconstruct the entire global scene, etc.
  • the AR function of the HUD may also be tested in the global scene subsequently, which is not specifically limited in this embodiment of the present application. In this way, through the above steps, a large number of various virtual simulation scenes and real scene simulation scenes can be generated based on the software method, thereby providing basic environment support for the AR imaging effect test of the HUD.
  • Step S902 fuse X first AR images corresponding to the X first outside scenes in the X first outside scenes to generate corresponding X second scenes; each first AR image includes: One or more AR icons.
  • step S902 reference may be made to step S802 and step S803 in the above-mentioned embodiment corresponding to FIG. 6a, and details are not repeated here.
  • the preset model may be a neural network model
  • the neural network model may be obtained by training from multiple scenes, multiple AR icons, and different matching degrees of multiple scenes and multiple AR icons.
  • the neural network model can be obtained by the following exemplary training method: taking multiple scenes as training samples, superimposing one or more AR icons matching the multiple scenes to obtain a plurality of corresponding augmented reality scenes, using the Multiple augmented reality scenarios are targeted and trained through deep learning algorithms to obtain results close to the target and obtain corresponding neural network models.
  • the scene in the above training sample may be a picture captured by a camera, or a point cloud image scanned by a lidar, etc., which is not specifically limited in this embodiment of the present application.
  • the computing device can simulate the HUD based on the preset HUD parameter set, and construct the X first outside scene scenes.
  • the corresponding X HUD virtual image surfaces wherein the construction of the HUD virtual image surface can also be realized by using the above-mentioned existing rendering engine technology.
  • the HUD parameter set may include at least one parameter among the curvature of the windshield, the position of the eye box, the observation position of the human eye, the installation position of the HUD, the size of the virtual image surface of the HUD, and the like.
  • FIG. 11 is a schematic diagram of a HUD virtual image surface provided by an embodiment of the present application. In Fig.
  • the first outside scene is taken as an example of a virtual simulation scene.
  • the HUD virtual image surface can be a corresponding area in the first outside scene.
  • Fig. 11 can be the HUD simulation effect from the driver's perspective, that is, It is equivalent to constructing a "screen" in front of the "driver"'s field of vision that matches the real HUD hardware parameters, and the subsequent AR function effects (that is, AR images) can be rendered and drawn on this "screen”. It can be implemented in a way that is not limited to off-screen rendering.
  • the driver's The position of the human eye should be in the position of the eye box calibrated by the HUD, that is, the position of the human eye is in the known best observation position, and the driver observes the object on the virtual image surface (that is, the AR image) at this position is the clearest and best. of.
  • a HUD virtual image surface with a distance of l, a height of h, a width of w, and a transmittance of alpha (alpha) can be constructed in the first exterior scene.
  • the surface can fully undertake the simulation function of the HUD.
  • the camera model can also be used to simulate real human eye observation at the eye box position.
  • the aforementioned first outside scene and the AR imaging effect of the virtual HUD (eg, the first AR image) can be imaged at the observation position.
  • the corresponding image frame can be cropped for use in the memory buffer according to the size of the HUD virtual image surface, and the first AR corresponding to the first scene outside the vehicle generated based on the preset model is used.
  • the image is rendered into the virtual image surface of the HUD to generate the corresponding second scene.
  • certain preprocessing such as cropping, scaling, and rotation, may be performed on the first AR image, so that the first AR image and The first outside scene and the HUD virtual image surface are adapted.
  • FIG. 12a is a schematic diagram of an AR imaging effect provided by an embodiment of the present application.
  • the second scene may inherit multiple scene elements from the first scene outside the vehicle.
  • the second scene includes a HUD virtual image surface and a plurality of AR icons (for example, a HUD virtual image surface) displayed on the HUD virtual image surface. It can include current speed information and current battery information as shown in Figure 12a, warning information of vehicles, sidewalks and traffic lights, as well as straight AR navigation guidance arrows, etc.).
  • FIG. 12b is a schematic diagram of another AR imaging effect provided by an embodiment of the present application.
  • the second scene is a real scene, including scene elements such as lanes, vehicles, and plants, as well as the constructed HUD virtual image surface and multiple AR icons in the HUD virtual image surface (in Figure 12b, the For example, the target box of the vehicle can be used to prompt the driver to pay attention to the vehicle next to the lane, so as to ensure driving safety, etc.).
  • FIG. 12c is a schematic diagram of another AR imaging effect provided by an embodiment of the present application.
  • the second scene may be a scene constructed based on point cloud data, which may also include multiple scene elements, a HUD virtual image surface, an AR icon, and the like.
  • step S902 may be as follows:
  • the projection v' of the spatial position of the object in the camera model (that is, the human eye) can be calculated using the following projection transformation relationship:
  • the projection matrix is: Among them, near is the near plane, far is the far plane, and top is the top.
  • the view matrix is:
  • the camera field of view (Fov) is: Among them, L h is the height of the camera imaging near plane, and f y is the focal length of the camera.
  • the AR icon to be rendered can be drawn on the accurate computing device screen coordinates.
  • FIG. 13 is a schematic diagram of a HUD simulation effect from an external perspective provided by an embodiment of the present application.
  • the HUD imaging process in the actual driving process as shown in Figure 13 can be completely simulated without considering any HUD distortion, human eye position changes and vehicle shake factors.
  • the above-mentioned camera field of view can be divided into a vertical field of view and a horizontal field of view.
  • the vertical field of view is the angle formed by the top and bottom of the HUD virtual image surface
  • the horizontal field of view The angle is the angle formed by the left and right edges of the HUD virtual image surface, and will not be described in detail here.
  • Step S903 analyze the AR effect based on the one or more scene elements and the one or more AR icons in each of the second scenes.
  • the computing device After the computing device generates the second scene, based on the positional relationship and/or logical relationship between one or more scene elements and one or more AR icons in each second scene, the current AR The effect is analyzed, and further, the preset model and a series of software algorithms can be corrected accordingly, so as to continuously optimize the AR function of the HUD and ensure the user experience.
  • the preset model and a series of software algorithms can be corrected accordingly, so as to continuously optimize the AR function of the HUD and ensure the user experience.
  • FIG. 14 is a schematic diagram of another AR imaging effect provided by an embodiment of the present application.
  • the position of the AR navigation guidance arrow shown in FIG. 14 obviously deviates from the position of the scene element (straight road), which does not fit the road surface.
  • testers can intuitively locate the possible problems and deficiencies of the current algorithm based on the AR imaging effect obtained by the current simulation, so as to improve the algorithm more accurately, efficiently and in a targeted manner.
  • the imaging effect of the HUD is affected by various factors.
  • parameters such as distortion, virtual image distance, and eyebox position of the HUD often need to be calibrated.
  • the installation position of the HUD is often deviated due to the problem of assembly accuracy, which in turn causes the virtual image distance (VID) of the HUD to change, resulting in AR images projected by the HUD (such as AR navigation guidance). arrows, etc.) produce distortion.
  • VID virtual image distance
  • the embodiment of the present application can directly perform distortion transformation on the memory value of the rendering buffer in the scene by extending support for HUD parameterization (that is, the AR image to be rendered). Distortion processing), real-time rendering to intuitively simulate the impact of parameter changes on AR imaging effects. In this way, the specific effects of the deviation of the HUD installation position and the reduction of AR rendering accuracy caused by distortion factors such as non-standard windshield curvature can be intuitively evaluated.
  • the Q first AR images corresponding to the Q first off-vehicle scenes may be distorted by a computing device to obtain Q corresponding second AR images; the Q first off-vehicle scenes and the Q first off-vehicle scenes may be fused. a first AR image to obtain the corresponding Q third scenes, and enable the display screen to display the Q third scenes.
  • each second AR image may also include one or more AR icons.
  • the distortion processing may include at least one of radial distortion, tangential distortion, virtual image distance increase and virtual image distance reduction, and Q is an integer greater than or equal to 1 and less than or equal to X.
  • FIG. 15 is a schematic diagram of a distortion type and AR imaging effect provided by an embodiment of the present application.
  • Various types of distortion processing and the resulting AR imaging effects are shown in Figure 15.
  • the obtained AR image has obvious distortion, which seriously affects the user's visual experience and use experience, and greatly reduces the driving comfort.
  • the displayed AR image will produce a certain degree of out-of-focus blur.
  • FIG. 16 is a schematic diagram for comparison of an AR imaging effect provided by an embodiment of the present application.
  • the AR imaging effect after multiple polynomial distortions is significantly reduced.
  • it can be superimposed in the frame buffer to be output to the rendering pipeline according to the pixel position correspondence of the distortion function or the distortion table, and can be superimposed multiple times according to the complexity of the distortion, and the AR imaging effect after the distortion processing can be intuitive.
  • the embodiment of the present application is further equivalent to providing a platform for developers to analyze the distortion function, so as to conduct research and development on the de-distortion function of the HUD, continuously improve the performance of the HUD, and ensure the use of the user. experience.
  • the de-distortion algorithm in the HUD needs to be tested, then further based on the de-distortion algorithm, the Q second AR images can be processed.
  • De-distortion processing to obtain the corresponding Q third AR images; fuse the Q first outside scenes and the Q third AR images to obtain the corresponding Q fourth scenes, and enable the display screen to display the Q the fourth scene.
  • the AR imaging effects before and after the de-distortion processing are compared, the de-distortion effect of the de-distortion algorithm is analyzed, and the de-distortion algorithm can be further modified accordingly.
  • any parameter change of the de-distortion model can obtain the corresponding effect in a WYSIWYG manner, which greatly improves the de-distortion function. Efficiency in development and testing.
  • the jitter during driving will also bring about the degradation of AR accuracy.
  • the jitter factors can generally include the jitter change of the driver's eye position and the HUD jitter caused by the driving process of the vehicle. icons) and real objects (that is, objects in the real world, such as road surfaces, pedestrians, and vehicles) have certain alignment distortions.
  • the heights of different drivers are different, which can easily lead to deviations between the observation position of individual drivers and the position of the eye box, so that the observed AR icons cannot fit the real-world objects, causing visual distortion and affecting the AR experience.
  • the development of the HUD anti-shake function is particularly important.
  • FIG. 17 is a schematic diagram of a human eye line of sight coordinate system and a HUD virtual image plane coordinate system provided by an embodiment of the present application.
  • the human eye sight coordinate system and the HUD virtual image surface coordinate system can both be established based on a fixed vehicle body coordinate system.
  • the human eye sight coordinate system can also be used as the vehicle body coordinate system, or the HUD virtual image surface coordinate system can be used as the vehicle body.
  • a coordinate system, etc., are not specifically limited in this embodiment of the present application.
  • FIG. 18 is a schematic diagram illustrating the influence of a shaking condition on an AR imaging effect provided by an embodiment of the present application.
  • the human eye position has displacement and the HUD virtual image surface has jitter, there will be a certain degree of alignment distortion between the observed AR image and the scene, that is, the AR icon and the corresponding scene.
  • the S first AR images corresponding to the S first off-vehicle scenes may also be shaken by a computing device to obtain S corresponding second AR images; the S first off-vehicle scenes and the S first AR images are obtained to obtain S corresponding third scenes, and the display screen is enabled to display the S third scenes.
  • the shaking processing may include superimposing a preset rotational displacement amount and/or shaking amount, and S is an integer greater than or equal to 1 and less than or equal to X.
  • the posture of the human eye can usually be abstracted as a six-dimensional vector. After the body coordinate system is fixed, the vector of the human eye relative to the origin of the coordinate system can be represented.
  • [t x , ty y , t z ] can represent the position offset of the human eye relative to the origin of the body coordinate system,
  • [ ⁇ x , ⁇ y , ⁇ z ] can represent the deflection angle [yaw (yaw angle), pitch (pitch angle), yoll (roll angle)] of the sight line of the human eye relative to the axes of the coordinate system.
  • the above quantities can be represented by the following transition matrix:
  • the transition matrix can correspond to the various quantities in the viewing angle matrix in the above-mentioned scene construction process. After a series of conversions, the viewing angle matrix of the human eye line of sight under different conditions can be calculated, and then the correct observation viewing angle of the virtual HUD can be obtained. It is understandable that, for the amount of shaking of the HUD, since there is usually only up and down shaking during the driving process, this variable can be idealized during the analysis process, the change of six degrees of freedom can be abstracted as the amount of shaking up and down, and the shaking The amount is added to the above transition matrix as a feedback amount, and finally reflected in the viewing angle matrix.
  • the rendering engine can adjust the corresponding observation effect according to the change of the viewing angle matrix, that is, different AR imaging effects can be obtained, so that the AR imaging effects can be intuitively targeted.
  • the changes of HUD are analyzed to support the follow-up research and development of the de-distortion function of the HUD, so as to continuously improve the performance of the HUD and ensure the user experience.
  • the above-mentioned S second The AR images are subjected to anti-shake processing to obtain the corresponding S third AR images; the S first exterior scenes and the S third AR images are fused to obtain the corresponding S fourth scenes, and the display is enabled.
  • the screen displays the S fourth scenes.
  • the AR imaging effects before and after the anti-shake processing are compared, the anti-shake effect of the anti-shake algorithm is analyzed, and the anti-shake algorithm can be further analyzed. Corresponding corrections are made to continuously improve the anti-shake function of the HUD.
  • the embodiment of the present application can also perform distortion processing and dither processing on the first AR image at the same time, so as to intuitively obtain the distorted image.
  • the AR imaging effect under the simultaneous influence of jitter and jitter correspondingly, the second AR image can also be de-distorted and jittered at the same time to analyze the de-distortion effect and anti-shake effect under the simultaneous influence of distortion and jitter, etc. etc., which are not specifically limited in the embodiments of the present application.
  • a simulation scene can be constructed by using an existing rendering engine or a real scene simulation scene can be constructed by using real road data collected in advance, and a virtual HUD screen (that is, a HUD screen) can be constructed.
  • Virtual image plane in the scene, draw AR icons in the HUD virtual image plane, so that the correctness of the AR function can be verified intuitively, providing a priori guarantee for driving safety and comfort. Since the scene can be arbitrarily synthesized and reused, the test coverage is greatly increased, which solves the problems of difficult restoration and non-repeatability of the AR function experience and test scene in the prior art to a certain extent.
  • the embodiments of the present application can support the repeatability of multiple road conditions, multiple weather, multiple regions, and multiple scenarios, get rid of the constraints of hardware devices, simplify the experience and testing process of AR functions, and facilitate the rapid development of models and algorithms. iterate.
  • the AR effect test for HUD mainly adopts offline calibration measurement, and the algorithm function is tested and improved by driving a real car, which largely relies on the test collection of offline data, so the AR function cannot be restored. Scenarios that fail (e.g. not displaying the correct AR icon or distorting the entire AR image).
  • the parameters have been basically determined.
  • the scenario processing method provided by the embodiment of the present application can greatly improve the test efficiency, has the advantages of wide coverage, high reusability, and low cost, and can support the expansion requirements of incremental scenarios and parameter introduction.
  • virtual simulation technology can be applied to many aspects, and its core idea is that everything about the real world can be constructed in a virtual three-dimensional environment. Therefore, in some possible embodiments, in addition to virtual AR-HUD Hardware equipment, in addition to simulating the display effect of AR function events triggered during driving on the HUD equipment, it can also simulate various scenarios and other equipment based on different requirements, so as to achieve efficient and low-cost testing and analysis of other equipment, and many more.
  • FIG. 19 is a schematic structural diagram of a scene processing apparatus provided by an embodiment of the present application.
  • the scene processing apparatus 30 may be applied to the above computing device.
  • the scene processing apparatus 30 may include a first acquisition unit 301, a second acquisition unit 302, a fusion unit 303 and a first display unit 304, wherein the detailed description of each unit is as follows:
  • the first acquiring unit 301 is configured to acquire a first scene outside the vehicle; wherein, the first scene outside the vehicle is a two-dimensional scene image or a three-dimensional scene model;
  • a second acquiring unit 302 configured to acquire a first AR image corresponding to the first scene outside the vehicle
  • a fusion unit 303 configured to fuse the first outside scene and the first AR image to obtain a second scene; wherein the first outside scene is the real information in the second scene, and the The first AR image is virtual information in the second scene;
  • the first display unit 304 is configured to enable the display screen to display the second scene.
  • the second obtaining unit 302 is specifically configured to:
  • the first AR image corresponding to the first outside scene is acquired according to the first outside scene and the preset model; wherein, the first AR image includes one or more AR icons.
  • the preset model is a neural network model
  • the neural network model is based on multiple scenes, multiple AR icons, and differences between the multiple scenes and the multiple AR icons
  • the matching degree is obtained by training.
  • the fusion unit 303 is specifically configured to:
  • a corresponding HUD virtual image surface is determined in the first off-vehicle scene; the HUD virtual image surface is a corresponding area in the first off-vehicle scene;
  • the first AR image is rendered into the HUD virtual image surface to obtain a second scene.
  • the HUD parameter set includes at least one parameter among the curvature of the windshield, the position of the eye box, the observation position of the human eye, the installation position of the HUD, and the size of the HUD virtual image surface.
  • the first obtaining unit 301 is specifically configured to:
  • Acquire data collected by a first sensor is a vehicle-mounted sensor; the data collected by the first sensor is data collected for the surrounding environment of the target vehicle during the driving process of the target vehicle, including image data, point clouds at least one of data, temperature data, and humidity data; the first sensor includes at least one of a camera, a lidar, a millimeter-wave radar, a temperature sensor, and a humidity sensor;
  • the first outside scene is constructed, and the first outside scene is a real-world simulation scene.
  • the first obtaining unit 301 is specifically configured to:
  • the second sensor is a sensor constructed by a preset simulation system
  • the data collected by the second sensor is the data set by the preset simulation system, including weather, Data of at least one of roads, pedestrians, vehicles, plants and traffic signals;
  • the first outside scene is constructed, and the first outside scene is a virtual simulation scene.
  • the apparatus 30 further includes:
  • the first preprocessing unit 305 is configured to perform a first preprocessing on the first AR image to obtain a second AR image corresponding to the first scene outside the vehicle; the first preprocessing includes distortion processing and jitter processing At least one of; the distortion processing includes at least one of radial distortion, tangential distortion, virtual image distance increase and virtual image distance reduction; the jitter processing includes superimposing a preset rotational displacement amount and/or jitter amount ;
  • a second fusion unit 306, configured to fuse the first outside scene and the second AR image to obtain a third scene
  • the second display unit 307 is configured to enable the display screen to display the third scene.
  • the apparatus 30 further includes:
  • the second preprocessing unit 308 is configured to perform a second preprocessing on the second AR image to obtain a third AR image corresponding to the first scene outside the vehicle; the second preprocessing includes dewarping processing and anti-aliasing at least one of dither processing;
  • a third fusion unit 309 configured to fuse the first outside scene and the third AR image to obtain a fourth scene
  • the third display unit 310 is configured to enable the display screen to display the fourth scene.
  • the apparatus 30 further includes:
  • An optimization unit 311, configured to obtain the processing effect of the de-distortion processing and/or the anti-shake processing based on the third scenario and the fourth scenario, so as to optimize the corresponding de-distortion function and/or anti-shake processing move function.
  • the first outside scene includes one or more scene elements; the one or more scene elements include one or more of weather, roads, pedestrians, vehicles, plants and traffic signals each; the one or more AR icons include one or more of left-turn, right-turn and straight navigation signs; the device 30 further includes:
  • a correction unit 312 configured to modify the preset model based on the positional relationship and/or logical relationship between the one or more scene elements and the one or more AR icons in each of the second scenes Make appropriate corrections.
  • the first obtaining unit 301 is configured to execute step S801 in the method embodiment corresponding to FIG. 6a; the second obtaining unit 302 is configured to execute step S802 in the method embodiment corresponding to FIG. 6a; the fusion unit 303 is configured to execute step S802 described above 6a corresponds to step S8/3 in the method embodiment; the first display unit 304 is configured to execute step S804 in the above-mentioned method embodiment corresponding to FIG. 6a; the first preprocessing unit 305, the second fusion unit 306, and the second display unit 307 , the second preprocessing unit 308, the third fusion unit 309, the third display unit 310, the optimization unit 311 and the correction unit 312 are used to perform step S804 in the corresponding method embodiment of FIG. 6a;
  • Each unit in FIG. 19 may be implemented in software, hardware, or a combination thereof.
  • Units implemented in hardware may include circuits and electric furnaces, algorithm circuits or analog circuits, and the like.
  • a unit implemented in software may include program instructions, is regarded as a software product, is stored in a memory, and can be executed by a processor to implement relevant functions, see the previous introduction for details.
  • FIG. 20 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
  • the computing device 1000 may be the computing device 100 in the above-mentioned FIGS. 3 a , 3 b and 3 c , wherein the computing device 1000 at least includes a processor 1001 , an input device 1002 , an output device 1003 , and a computer-readable storage medium 1004, the database 1005 and the memory 1006, the computing device 1000 may also include other general components, which will not be described in detail here.
  • the processor 1001, the input device 1002, the output device 1003, and the computer-readable storage medium 1004 in the computing device 1000 may be connected by a bus or other means.
  • the processor 1001 can be used to implement the first acquisition unit 301 , the second acquisition unit 302 , the fusion unit 303 , the first display unit 304 , the first preprocessing unit 305 , the second fusion unit 306 , the second The display unit 307 , the second preprocessing unit 308 , the third fusion unit 309 , the third display unit 310 , the optimization unit 311 and the correction unit 312 , wherein, for the technical details of the implementation process, refer to the method described in FIG. 6 a above.
  • steps S801 to S803 in the embodiment you can also refer to the relevant descriptions of steps S901 to S903 in the method embodiment described in FIG.
  • the processor 1001 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the above programs.
  • CPU central processing unit
  • ASIC application-specific integrated circuit
  • the memory 1006 within the computing device 1000 may be read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types of static storage devices that can store Other types of dynamic storage devices for information and instructions, which may also be Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or Other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or capable of being used to carry or store desired in the form of instructions or data structures Program code and any other medium that can be accessed by a computer, but is not limited thereto.
  • the memory 1006 may exist independently and be connected to the processor 1001 through a bus.
  • the memory 1006 may also be integrated with the processor 1001.
  • a computer readable storage medium 1004 may be stored in the memory 1006 of the computing device 1000, the computer readable storage medium 1004 for storing a computer program comprising program instructions, the processor 1001 for executing the computer Readable storage medium 1004 stores program instructions.
  • the processor 1001 (or called CPU (Central Processing Unit, central processing unit)) is the computing core and the control core of the computing device 1000, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to thereby Implement corresponding method processes or corresponding functions; in one embodiment, the processor 1001 described in this embodiment of the present application may be used to acquire a first outside scene; wherein, the first outside scene is a two-dimensional scene image or a three-dimensional scene model; obtaining a first AR image corresponding to the first outside scene; fusing the first outside scene and the first AR image to obtain a second scene; wherein the first scene The outside scene is the real information in the second scene, the first AR image is the virtual information in the second scene; the display screen is enabled to display
  • Embodiments of the present application further provide a computer-readable storage medium (Memory), where the computer-readable storage medium is a memory device in the computing device 1000 for storing programs and data.
  • the computer-readable storage medium here may include both a built-in storage medium in the computing device 1000 , and certainly also an extended storage medium supported by the computing device 1000 .
  • the computer-readable storage medium provides storage space in which the operating system of the computing device 1000 is stored.
  • one or more instructions suitable for being loaded and executed by the processor 1001 are also stored in the storage space, and these instructions may be one or more computer programs (including program codes).
  • the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory;
  • a computer-readable storage medium for the processor can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory; A computer-readable storage medium for the processor.
  • the embodiments of the present application also provide a computer program, the computer program includes instructions, when the computer program is executed by the computer, the computer can execute part or all of the steps of any scene processing method.
  • the disclosed apparatus may be implemented in other manners.
  • the device embodiments described above are only illustrative.
  • the division of the above-mentioned units is only a logical function division.
  • multiple units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical or other forms.
  • the units described above as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the integrated units are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium.
  • the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc., specifically a processor in the computer device) to execute all or part of the steps of the above methods in various embodiments of the present application.
  • a computer device which may be a personal computer, a server, or a network device, etc., specifically a processor in the computer device
  • the aforementioned storage medium may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (Read-Only Memory, abbreviation: ROM) or random access memory (Random Access Memory, abbreviation: RAM) and other various storage media that can store medium of program code.
  • a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer.
  • the application running on the end device and the end device can be components.
  • One or more components may reside within a process and/or thread of execution, and a component may be localized on one computer and/or distributed between 2 or more computers.
  • these components can execute from various computer readable media having various data structures stored thereon.
  • a component may, for example, be based on a signal having one or more data packets (eg, data from two components interacting with another component between a local system, a distributed system, and/or a network, such as the Internet interacting with other systems via signals) Communicate through local and/or remote processes.
  • data packets eg, data from two components interacting with another component between a local system, a distributed system, and/or a network, such as the Internet interacting with other systems via signals

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Computer Hardware Design (AREA)
  • Biophysics (AREA)
  • Computer Graphics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Traffic Control Systems (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请实施例公开了一种场景处理方法、装置、系统及相关设备，该方法包括：获取第一车外场景；其中，所述第一车外场景为二维的场景图像或者三维的场景模型；获取与所述第一车外场景对应的第一AR图像；融合所述第一车外场景和所述第一AR图像，以获得第二场景；其中，所述第一车外场景为所述第二场景中的现实信息，所述第一AR图像为所述第二场景中的虚拟信息；使能显示屏显示所述第二场景。采用本申请实施例可以更加高效、便捷地融合现实信息与虚拟信息，从而得到直观的AR效果。

Description

一种场景处理方法、装置、系统及相关设备 技术领域
本申请实施例涉及计算机技术领域，尤其涉及一种场景处理方法、装置、系统及相关设备。
背景技术
抬头显示器(head up display，HUD)是一种将图像投射到驾驶员前方视野中的现实装置。相比于传统仪表和中控屏幕，驾驶员在观察HUD图像时，无需低头，避免了人眼焦距在图像和路面之间的来回切换，减少了危机反应的时间，提高了驾驶安全性。而近年来提出的增强现实抬头显示器(augmented realityhead up display，AR-HUD)，则可以进一步将HUD投射的图像与真实路面信息融合起来，实现增强现实(augmented reality，AR)导航和AR预警等各类功能，大大增强驾驶员对路面信息的获取，保证驾驶的安全性和舒适性。
然而，目前针对HUD的成像效果研究多集中于光学分析，若想要对利用车载HUD实现的AR增强现实功能(例如AR地图、AR导航和兴趣点(point of interest，POI)显示等)进行分析测试，则往往需要在HUD安装标定之后，才能观测到实际的AR投影效果。因此，目前为了验证HUD的AR成像效果，不断优化AR功能，大多采取实车测试的方法，也即将装载有AR-HUD的车辆驾驶到现实环境中进行测试，需要耗费大量的时间资源以及人力物力，成本高，效率低。另外，由于现实中的场景时时刻刻在变化，导致大量测试场景无法复现，继而使得测试结果可靠性低，从而影响AR-HUD中涉及的一系列AR算法的开发和测试，无法有效保证用户的使用体验。
因此，如何高效、便捷地融合现实信息与虚拟信息，从而得到直观的AR效果是亟待解决的问题。
发明内容
本申请实施例提供一种场景处理方法、装置、系统及相关设备，可以高效、便捷地融合现实信息与虚拟信息，得到直观的AR效果。
本申请实施例提供的场景处理方法可以由电子装置等执行。电子装置是指能够被抽象为计算机系统的设备，其中，支持场景处理功能的电子装置，也可称为场景处理装置。场景处理装置可以是该电子装置的整机，例如：智能手机、平板电脑、笔记本电脑、台式电脑、车机、车载电脑或服务器等；也可以是由多个整机构成的车载系统/装置；还可以是该电子装置中的部分器件，例如：场景处理功能相关的芯片，如系统芯片或场景处理芯片其中，系统芯片也称为片上系统，或称为SoC芯片。具体地，场景处理装置可以是诸如智能车辆中车机、车载电脑等这样的终端装置，也可以是能够被设置在智能终端的计算机系统或环视系统中的系统芯片或场景处理芯片。
此外，本申请实施例提供的场景处理方法可以应用于如下场景：车载仿真系统、车载游戏、终端云看车、直播和车领域测试等等多种场景。
第一方面，本申请实施例提供了一种场景处理方法，该方法包括：获取第一车外场景；其中，所述第一车外场景为二维的场景图像或者三维的场景模型；获取与所述第一车外场景对应的第一AR图像；融合所述第一车外场景和所述第一AR图像，以获得第二场景；其中， 所述第一车外场景为所述第二场景中的现实信息，所述第一AR图像为所述第二场景中的虚拟信息；使能显示屏显示所述第二场景。
通过第一方面提供的方法，本申请实施例可以基于软件的方法，通过现有的计算设备(比如笔记本电脑和台式电脑等)构建大量二维、或三维的仿真场景(每个仿真场景均可包括如车辆、道路或行人等场景元素)；然后，基于上述仿真场景以及开发的模型，生成与上述仿真场景对应的AR图像，并将上述仿真场景与对应的AR图像进行融合，从而快速、高效得到大量包括现实信息和虚拟信息的增强现实场景。其中，每个AR图像均可包括AR导航指引箭头、AR预警等AR图标。如此，相较于现有技术，本申请实施例可以不依赖于现实场景以及HUD硬件设备，而是通过软件的方法快速构建大量可复现的场景，覆盖性广、重用性高。如此，不仅可以快速、高效地展示各类增强现实场景，后续还可以基于本申请实施例仿真得到的各类增强现实场景下直观的AR效果，不断优化和改进HUD的AR功能，保证用户的使用体验。
在一种可能的实现方式中，所述获取与所述第一车外场景对应的第一AR图像，包括：根据所述第一车外场景与预设模型，获取与所述第一车外场景对应的所述第一AR图像；其中，所述第一AR图像包括一个或多个AR图标。
在本申请实施例中，可以通过预先训练得到的模型生成与仿真场景相匹配的AR图像，以得到增强现实中的虚拟信息。例如，该场景包括一个十字路口，基于导航信息，当前十字路口应该右转，则可生成相应的右转指引箭头，又例如，该场景为文化景点，则可以生成对应的景点指示信息和景点介绍等等。
在一种可能的实现方式中，所述预设模型为神经网络模型，所述神经网络模型是根据由多个场景、多个AR图标以及所述多个场景和所述多个AR图标的不同匹配度训练得到的。
在本申请实施例中，可以通过多个场景、多个AR图标以及所述多个场景和所述多个AR图标的不同匹配度预先对神经网络模型进行训练，从而后续可以基于该神经网络模型对该大量仿真场景进行识别，并生成与之对应的AR图像，以快速、高效地得到大量增强现实场景。
在一种可能的实现方式中，所述融合所述第一车外场景和所述第一AR图像，以获得第二场景，包括：基于预设的抬头显示器HUD参数集合，在所述第一车外场景中确定对应的HUD虚像面；所述HUD虚像面为所述第一车外场景中对应的一个区域；将所述第一AR图像渲染至所述HUD虚像面中，以获得第二场景。
在本申请实施例中，可以基于软件的方法，通过现有的计算设备对车载HUD硬件设备进行仿真，从而在该大量场景中确定对应的虚像面，该虚像面可以为场景中的一个区域。可以理解的是，虚像面即为HUD投射出的用于显示AR图像的面。然后，通过预先训练得到的模型生成与场景相匹配的AR图像，并将AR图像渲染至该虚像面中。如此，本申请实施例可以通过软件的方法，仿真模拟各类场景下HUD投射出的AR图像，完成对AR功能(该功能例如包括上述用于识别场景并生成对应AR图像的预设模型)的测试，以及。大大节省实车测试所需消耗的时间资源以及人力物力，提高AR功能的测试效率，进一步保证HUD的用户使用体验，提升用户的驾驶舒适性和安全性。
在一种可能的实现方式中，所述HUD参数集合包括挡风玻璃曲率、眼盒位置、人眼观测位置、HUD安装位置和HUD虚像面尺寸中的至少一种参数。
在本申请实施例中，可以基于HUD的相关硬件参数，例如挡风玻璃曲率、眼盒位置、人眼观测位置、HUD安装位置和HUD虚像面尺寸等等，通过软件的方法对HUD进行仿真， 以在大量的场景中构建对应的HUD虚像面，为后续的AR功能测试提供支持。如此，本申请实施例可以不依赖于硬件设备，从而节省了测试人员外出进行实车测试的人力和物力，并且通过软件仿真HUD的方法可以更加高效和低成本的对HUD中的相关AR功能等进行覆盖率更高的测试，有效的支持AR功能的开发和测试工作，大大提高了测试效率和测试结果的可靠性，从而可以对HUD中一系列软件算法进行更好的改善优化，进一步保证用户的使用体验，提升用户的驾驶舒适性和安全性。
在一种可能的实现方式中，所述获取第一车外场景，包括：获取第一传感器采集的数据；所述第一传感器为车载传感器；所述第一传感器采集的数据为在目标车辆行驶过程中针对所述目标车辆的周围环境采集的数据，包括图像数据、点云数据、温度数据和湿度数据中的至少一种；所述第一传感器包括摄像头、激光雷达、毫米波雷达、温度传感器和湿度传感器中的至少一种；基于所述第一传感器采集的数据，构建所述第一车外场景，所述第一车外场景为实景仿真场景。
在本申请实施例中，只需通过驾驶安装有车载传感器(例如摄像头、激光雷达、毫米波雷达等)的车辆去实际道路中，通过该一个或多个车载传感器采集周围环境的数据。如此，便可以通过软件的方法，在电脑等计算设备中基于采集到的大量数据构建上述大量仿真场景(例如可以为由激光雷达采集到的点云数据构建而成的仿真场景，也可以是由摄像头采集到图像数据构建而成的仿真场景，或者是融合激光雷达和摄像机各自采集到的数据后构建而成的仿真场景等等，本申请实施例对此不作具体限定)。显然，该大量仿真场景为实景仿真场景。如上所述，该大量仿真场景可以用于支持后续的AR功能展示以及测试等，大大提高了场景的覆盖率，并且保证了场景的可复现，从而保证后续可以对改进后的AR功能在相同场景下再次进行测试，已验证改进后的AR效果等等，有效提升测试结果的可靠性。
在一种可能的实现方式中，上述获取目标场景集合，包括：所述获取第一车外场景，包括：获取第二传感器采集的数据；所述第二传感器为通过预设的仿真模拟系统构建的传感器；所述第二传感器采集的数据为通过所述预设的仿真模拟系统设置的数据，包括天气、道路、行人、车辆、植物和交通信号中的至少一种数据；基于所述第二传感采集的数据，构建所述第一车外场景，所述第一车外场景为虚拟仿真场景。
在本申请实施例中，还可以通过软件的方法在电脑等计算设备上仿真得到多个虚拟传感器，并设置各个虚拟传感器的数据，从而可以基于该多个虚拟传感器的数据构建上述大量仿真场景，显然，该大量仿真场景为虚拟仿真场景(类似于构建虚拟的游戏场景)。如此，可以进一步节省测试人员外出进行实车驾驶，以获取车辆内多个传感器采集的数据所需消耗的时间资源以及人力、物力等等，进一步降低了测试成本。如上所述，该大量仿真场景可以用于支持后续的AR功能展示以及测试，大大提高了场景的覆盖率，并且保证了场景的可复现，从而保证后续可以对改进后的AR功能在相同场景下再次进行测试，已验证改进后的AR效果等等，有效提升测试结果的可靠性。
在一种可能的实现方式中，上述方法还包括：对所述第一AR图像进行第一预处理，获取与所述第一车外场景对应的第二AR图像；所述第一预处理包括畸变处理和抖动处理中的至少一种；所述畸变处理包括径向畸变、切向畸变、虚像距增大和虚像距减小中的至少一种；所述抖动处理包括叠加预设的旋转位移量和/或抖动量；融合所述第一车外场景和所述第二AR图像，以获得第三场景；使能所述显示屏显示所述第三场景。
在本申请实施例中，可以理解的是，由于实际在车辆上安装HUD的过程中往往会由于 安装精度的问题，使得HUD的安装位置与预设的理想位置之间存在偏差，或者由于生产制造精度的问题使得汽车挡风玻璃的曲率不符合标准，从而造成HUD投射出的AR图像存在或多或少的畸变，严重影响用户的视觉感受，极大程度上降低用户的使用体验以及驾驶舒适性，甚至危害用户的驾驶安全。因此，本申请实施例进一步地，还可以通过软件的方法对各类场景中生成的AR图像进行畸变处理，例如包括径向畸变、切向畸变、虚像距增大和虚像距减小等，覆盖范围广，效率高，从而可以直观地了解各类畸变对AR成像效果的影响，以对各类畸变造成的AR成像效果退化进行补偿，例如可以为后续去畸变功能的开发和测试提供有效、精确的支持，提升用户的使用体验。另外，当HUD进行实车装配后，由于车辆在行驶过程中总会遇到一定程度的颠簸，从而导致驾驶员的人眼位置或者HUD的位置发生抖动变化，相应的，驾驶员观测到的AR图像与真实场景中的物体(例如道路和车辆等)之间便会存在一定的对齐失真，不再贴合，从而影响用户的视觉感受。现有技术中，往往需要驾驶实车到多种路面上，以对抖动情况下的AR成像效果进行测试分析，但是由于实车测试过程中往往会伴随有多种抖动噪声，也即可能同时存在人眼抖动和HUD位置抖动等，导致难以抽离单个抖动因素进行测试建模分析，进而为后续防抖动功能的开发和测试带来困难，无法保证用户在实际驾驶中的使用体验。因此，相较于现有技术，本申请实施例进一步地，还可以通过软件的方法对各类场景中生成的AR图像进行抖动处理，例如可以根据实际需求依次叠加不同程度的旋转位移量和抖动量等，或者单独叠加一定的旋转位移量或者抖动量，等等，从而可以抽离单个抖动因素对AR成像效果进行测试分析，直观的了解在各类抖动情况对AR成像效果的影响，从而为后续防抖动功能的开发和测试提供有效、精确的支持，提升用户的使用体验。
在一种可能的实现方式中，上述方法还包括：对所述第二AR图像进行第二预处理，获取与所述第一车外场景对应的第三AR图像；所述第二预处理包括去畸变处理和防抖动处理中的至少一种；融合所述第一车外场景和所述第三AR图像，以获得第四场景；使能所述显示屏显示所述第四场景。
在一种可能的实现方式中，上述方法还包括：基于所述第三场景和所述第四场景，获取所述去畸变处理和/或所述防抖动处理的处理效果，以优化相应的去畸变功能和/或防抖动功能。
在本申请实施例中，进一步地，在进行畸变处理后，还可以对畸变处理后的AR图像叠加待测试的去畸变算法，从而可以直观的了解到当前的去畸变效果，以对去畸变算法不断进行相应的改进。继而保证在HUD安装位置存在偏移或者挡风玻璃曲率不合适的情况下的AR成像效果，保证用户的视觉感受和使用体验，进一步提升用户的驾驶舒适性和安全性。进一步地，在进行抖动处理后，还可以对抖动处理后的AR图像叠加待测试的防抖动算法，从而可以直观的了解到当前防抖动算法的防抖动效果，以对防抖动算法不断进行相应的改进。继而保证在实际驾驶中车辆颠簸造成人眼位置抖动或者HUD位置抖动情况下的AR成像效果，保证用户的视觉感受和使用体验，进一步提升用户的驾驶舒适性和安全性。
在一种可能的实现方式中，所述第一车外场景包括一个或多个场景元素；所述一个或多个场景元素包括天气、道路、行人、车辆、植物和交通信号中的一个或多个；所述一个或多个AR图标包括左转、右转和直行导航标识中的一个或多个；所述方法还包括：基于所述每个第二场景中的所述一个或多个场景元素以及所述一个或多个AR图标之间的位置关系和/或逻辑关系，对所述预设模型进行相应的修正。
在本申请实施例中，每个仿真场景中均可以包括一个或多个场景元素，例如道路、行人、 车辆、植物和交通信号(比如红绿灯等)，还例如路标、立交桥、建筑和动物，等等，本申请实施例对此不作具体限定。如此，本申请实施例可以根据仿真场景中的一个或多个场景元素与生成的AR图像中的一个或多个AR图标之间的匹配度(例如包括双方的位置关系和/或逻辑关系)，对预设模型的AR效果进行分析，并对该预设模型进行相应的修正。例如，该仿真场景可以包括一个十字路口和道路旁的一些树木等，基于导航信息，当前十字路口应该右转，则需要生成相应的右转导航标识(即例如右转指引箭头)，若该AR图像中生成的右转指引箭头没有很好的贴合路面而是显示在道路左侧的树木上，或者生成了生成了错误的直行指引箭头等，则可以认为当前的AR效果不理想，场景元素与对应的AR图标之间的位置关系和/或逻辑关系没有很好的匹配，预设模型仍需要进行改进。如此，本申请实施例可以通过仿真模拟的方法，使得测试人员在计算设备上显著地观察得到当前预设模型的AR效果，或者说直观的掌握当前HUD的AR功能情况，从而可以高效、准确的定位其中存在的问题，以更好的对预设模型等进行改进优化，从而保证用户的使用效果，提升用户的驾驶安全性和舒适性。
第二方面，本申请实施例提供了一种场景处理装置，包括：
第一获取单元，用于获取第一车外场景；其中，所述第一车外场景为二维的场景图像或者三维的场景模型；
第二获取单元，用于获取与所述第一车外场景对应的第一AR图像；
融合单元，用于融合所述第一车外场景和所述第一AR图像，以获得第二场景；其中，所述第一车外场景为所述第二场景中的现实信息，所述第一AR图像为所述第二场景中的虚拟信息；
第一显示单元，用于使能显示屏显示所述第二场景。
在一种可能的实现方式中，所述第二获取单元，具体用于：
根据所述第一车外场景与预设模型，获取与所述第一车外场景对应的所述第一AR图像；其中，所述第一AR图像包括一个或多个AR图标。
在一种可能的实现方式中，所述预设模型为神经网络模型，所述神经网络模型是根据由多个场景、多个AR图标以及所述多个场景和所述多个AR图标的不同匹配度训练得到的。
在一种可能的实现方式中，所述融合单元，具体用于：
基于预设的抬头显示器HUD参数集合，在所述第一车外场景中确定对应的HUD虚像面；所述HUD虚像面为所述第一车外场景中对应的一个区域；
将所述第一AR图像渲染至所述HUD虚像面中，以获得第二场景。
在一种可能的实现方式中，所述HUD参数集合包括挡风玻璃曲率、眼盒位置、人眼观测位置、HUD安装位置和HUD虚像面尺寸中的至少一种参数。
在一种可能的实现方式中，所述第一获取单元，具体用于：
获取第一传感器采集的数据；所述第一传感器为车载传感器；所述第一传感器采集的数据为在目标车辆行驶过程中针对所述目标车辆的周围环境采集的数据，包括图像数据、点云数据、温度数据和湿度数据中的至少一种；所述第一传感器包括摄像头、激光雷达、毫米波雷达、温度传感器和湿度传感器中的至少一种；
基于所述第一传感器采集的数据，构建所述第一车外场景，所述第一车外场景为实景仿真场景。
在一种可能的实现方式中，所述第一获取单元，具体用于：
获取第二传感器采集的数据；所述第二传感器为通过预设的仿真模拟系统构建的传感器；所述第二传感器采集的数据为通过所述预设的仿真模拟系统设置的数据，包括天气、道路、行人、车辆、植物和交通信号中的至少一种数据；
基于所述第二传感采集的数据，构建所述第一车外场景，所述第一车外场景为虚拟仿真场景。
在一种可能的实现方式中，所述装置还包括：
第一预处理单元，用于对所述第一AR图像进行第一预处理，获取与所述第一车外场景对应的第二AR图像；所述第一预处理包括畸变处理和抖动处理中的至少一种；所述畸变处理包括径向畸变、切向畸变、虚像距增大和虚像距减小中的至少一种；所述抖动处理包括叠加预设的旋转位移量和/或抖动量；
第二融合单元，用于融合所述第一车外场景和所述第二AR图像，以获得第三场景；
第二显示单元，用于使能所述显示屏显示所述第三场景。
在一种可能的实现方式中，所述装置还包括：
第二预处理单元，用于对所述第二AR图像进行第二预处理，获取与所述第一车外场景对应的第三AR图像；所述第二预处理包括去畸变处理和防抖动处理中的至少一种；
第三融合单元，用于融合所述第一车外场景和所述第三AR图像，以获得第四场景；
第三显示单元，用于使能所述显示屏显示所述第四场景。
在一种可能的实现方式中，所述装置还包括：
优化单元，用于基于所述第三场景和所述第四场景，获取所述去畸变处理和/或所述防抖动处理的处理效果，以优化相应的去畸变功能和/或防抖动功能。
在一种可能的实现方式中，所述第一车外场景包括一个或多个场景元素；所述一个或多个场景元素包括天气、道路、行人、车辆、植物和交通信号中的一个或多个；所述一个或多个AR图标包括左转、右转和直行导航标识中的一个或多个；所述装置还包括：
修正单元，用于基于所述每个第二场景中的所述一个或多个场景元素以及所述一个或多个AR图标之间的位置关系和/或逻辑关系，对所述预设模型进行相应的修正。
第三方面，本申请实施例提供了一种场景处理系统，包括：终端和服务器；
所述终端用于发送第一车外场景；其中，所述第一车外场景为所述终端的传感器获取的传感信息；
所述服务器用于接收来自于所述终端的所述第一车外场景；
所述服务器还用于获取与所述第一车外场景对应的第一AR图像；
所述服务器还用于融合所述第一车外场景和所述第一AR图像，以获得第二场景；其中，所述第一车外场景为所述第二场景中的现实信息，所述第一AR图像为所述第二场景中的虚拟信息；
所述服务器还用于发送所述第二场景；
所述终端还用于接收所述第二场景，并显示所述第二场景。
在一种可能的实现方式中，所述传感器包括温度传感器、湿度传感器、全球定位系统、摄像头和激光雷达中的至少一种；所述传感信息包括温度、湿度、天气、位置、图像和点云中的至少一种。
在一种可能的实现方式中，所述服务器，具体用于：
根据所述第一车外场景与预设模型，获取与所述第一车外场景对应的所述第一AR图像；所述第一AR图像包括一个或多个AR图标。
在一种可能的实现方式中，所述预设模型为神经网络模型，所述神经网络模型是根据由多个场景、多个AR图标以及所述多个场景和所述多个AR图标的不同匹配度训练得到的。
在一种可能的实现方式中，所述服务器，具体用于：
基于预设的抬头显示器HUD参数集合，在所述第一车外场景中确定对应的HUD虚像面；所述HUD虚像面为所述第一车外场景中对应的一个区域；
将所述第一AR图像渲染至所述HUD虚像面中，以获得第二场景。
在一种可能的实现方式中，所述HUD参数集合包括挡风玻璃曲率、眼盒位置、人眼观测位置、HUD安装位置和HUD虚像面尺寸中的至少一种参数。
在一种可能的实现方式中，所述服务器，还用于：
对所述第一AR图像进行第一预处理，获取与所述第一车外场景对应的第二AR图像；所述第一预处理包括畸变处理和抖动处理中的至少一种；所述畸变处理包括径向畸变、切向畸变、虚像距增大和虚像距减小中的至少一种；所述抖动处理包括叠加预设的旋转位移量和/或抖动量；
融合所述第一车外场景和所述第二AR图像，以获得第三场景；
发送所述第三场景；
所述终端还用于接收所述第三场景，并显示所述第三场景。
在一种可能的实现方式中，所述服务器，还用于：
对所述第二AR图像进行第二预处理，获取与所述第一车外场景对应的第三AR图像；所述第二预处理包括去畸变处理和防抖动处理中的至少一种；
融合所述第一车外场景和所述第三AR图像，以获得第四场景；
发送所述第四场景。
所述终端还用于接收所述第四场景，并显示所述第四场景。
在一种可能的实现方式中，所述服务器，还用于：
基于所述第三场景和所述第四场景，获取所述去畸变处理和/或所述防抖动处理的处理效果，以优化相应的去畸变功能和/或防抖动功能。
在一种可能的实现方式中，所述第一车外场景包括一个或多个场景元素；所述一个或多个场景元素包括天气、道路、行人、车辆、植物和交通信号中的一个或多个；所述一个或多个AR图标包括左转、右转和直行导航标识中的一个或多个；所述服务器，还用于：
基于所述每个第二场景中的所述一个或多个场景元素以及所述一个或多个AR图标之间的位置关系和/或逻辑关系，对所述预设模型进行相应的修正。
第四方面，本申请实施例提供了一种计算设备，该计算设备中包括处理器，处理器被配置为支持该计算设备实现第一方面提供的场景处理方法中相应的功能。该计算设备还可以包括存储器，存储器用于与处理器耦合，其保存该计算设备必要的程序指令和数据。该计算设备还可以包括通信接口，用于该计算设备与其他设备或通信网络通信。
其中，该计算设备可以是终端，例如手机、车机、车载PC等车载装置、汽车等交通工具，服务器。其中，服务器可以是虚拟服务器，也可以是实体服务器。还可以是芯片或电子系统等。
第五方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，该计算机程序被处理器执行时实现上述第一方面中任意一项所述的场景处理方法流程。其中，该处理器可以为一个或多个处理器。
第六方面，本申请实施例提供了一种计算机程序，该计算机程序包括指令，当该计算机程序被计算机执行时，使得计算机可以执行上述第一方面中任意一项所述的场景处理方法流程。
第七方面，本申请实施例提供了一种芯片系统，该芯片系统可以包括上述第二方面中任意一项所述的场景处理装置，用于实现上述第一方面中任意一项所述的场景处理方法流程所涉及的功能。在一种可能的设计中，所述芯片系统还包括存储器，所述存储器，用于保存场景处理方法必要的程序指令和数据。该芯片系统，可以由芯片构成，也可以包含芯片和其他分立器件。
第八方面，本申请实施例提供了一种电子装置，该电子装置可以包括上述第二方面中任意一项所述的场景处理装置，用于实现上述第一方面中任意一项所述的场景处理方法流程所涉及的功能。在一种可能的设计中，所述电子装置还包括存储器，所述存储器，用于保存场景处理方法必要的程序指令和数据。该电子装置，可以是终端，例如手机、车机、车载PC等车载装置、汽车等交通工具，服务器。其中，服务器可以是虚拟服务器，也可以是实体服务器。还可以是芯片或电子系统等。
综上，本申请实施例提供了一种场景处理方法，本申请实施例可以基于软件仿真的方法，通过现有的计算设备(比如手机、车机、服务器等)构建大量二维或三维的仿真场景(每个仿真场景均可包括如车辆、道路或行人等场景元素)。然后，基于开发的模型，生成与上述仿真场景对应的AR图像，并将上述仿真场景与对应的AR图像进行融合，从而快速、高效地得到大量包括现实信息和虚拟信息的增强现实场景。
下面，将结合几种较为常见的场景对本申请实施例的有益效果进行阐述。
首先，对于进行HUD相关产品售卖或者展示的商家来说，通过本申请实施例提供的一种场景处理方法，无需配备HUD硬件设备，也无需驾驶实车，只需通过商店内的显示屏便能直观地展示HUD的AR效果。相应的，客户也可以通过商店内的显示屏简单、直接地体验到HUD的AR效果。甚至，商家还可以将通过仿真得到的大量增强现实场景上传至云端，用户可以利用自己的手机通过相应的网站或者应用软件查看上述增强现实场景，从而更快速、便捷地了解产品的功能，以根据自身需求选购产品。如此，本申请实施例不仅为商家节省了大量人力物力，同时为客户提供了便捷。
此外，对于开发人员来说，可以基于每个融合了AR图像后的仿真场景中的场景元素和AR图标，快速、直观地对其中涉及的各类模型或者算法进行分析，以优化相应的功能。如此，相较于现有技术，本申请实施例可以不依赖于真实世界的场景以及HUD硬件设备，而是通过软件的方法快速构建各类模型或者算法测试所需的大量可复现的场景，覆盖性广、重用性高，进而可以在该大量场景中对各类模型或者算法进行测试，快速、直观地得到在各类场景下HUD投射出的AR图像，以不断优化AR功能以及相应的去畸变和防抖动功能等，保证用户的使用体验。由此，对于开发人员来说，本申请实施例不仅可以大大提高对HUD的AR功能的测试效率、场景覆盖率和测试结果的可靠性，还可以极大程度上节省实车测试所需消耗 的时间资源以及测试人员外出进行实车测试的人力和物力等等，大大减少了测试成本。
还有，对于实时直播的场景而言，例如在一些进行HUD相关产品推广的直播活动中，或者进行AR游戏的直播活动中。主播可以通过直播设备(例如手机终端，还可以包括其他的摄像机等)实时采集周围环境的信息(例如天气、道路、车辆和行人等)，并将该信息实时发送至服务器，服务器基于接收到的信息实时构建仿真场景，还原主播周围的真实环境，然后生成相应的AR图像并与上述实时的仿真场景进行融合，从而得到实时的增强现实场景，并发送给直播设备，由直播设备进行显示。如此，远程的观众可以利用手中的手机或者平板等设备通过网络直播间直观的感受实时的AR效果。可选地，上述实时直播的场景中，也可以不采用服务器，而是直接通过直播设备进行仿真场景的构建、AR图像的生成、融合以及增强现实场景的显示，等等，本申请实施例对此不作具体限定。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对本申请实施例或背景技术中所需要使用的附图进行说明。
图1是一种AR-HUD成像效果示意图。
图2是本申请实施例提供的一种智能车辆的功能框图。
图3a是本申请实施例提供的一种场景处理方法的系统架构示意图。
图3b是本申请实施例提供的另一种场景处理方法的系统架构示意图。
图3c是本申请实施例提供的一种计算设备的功能框图。
图4是本申请实施例提供的一种场景处理方法的应用场景示意图。
图5是本申请实施例提供的另一种场景处理方法的应用场景示意图。
图6a是本申请实施例提供的一种场景处理方法的流程示意图。
图6b是本申请实施例提供的另一种场景处理方法的流程示意图。
图7是本申请实施例提供的另一种场景处理方法的流程示意图。
图8是本申请实施例提供的一种场景的示意图。
图9是本申请实施例提供的一种相机坐标系和世界坐标系的示意图。
图10是本申请实施例提供的一种场景重构的示意图。
图11是本申请实施例提供的一种HUD虚像面的示意图。
图12a是本申请实施例提供的一种AR成像效果的示意图。
图12b是本申请实施例提供的另一种AR成像效果的示意图。
图12c是本申请实施例提供的又一种AR成像效果的示意图。
图13是本申请实施例提供的一种外部视角下的HUD模拟仿真效果示意图。
图14是本申请实施例提供的又一种AR成像效果的示意图。
图15是本申请实施例提供的一种畸变类型与AR成像效果的示意图。
图16是本申请实施例提供的一种AR成像效果的对比示意图。
图17是本申请实施例提供的一种人眼视线坐标系和HUD虚像面坐标系的示意图。
图18是本申请实施例提供的一种抖动情况对AR成像效果影响的示意图。
图19是本申请实施例提供的又一种场景处理装置的结构示意图。
图20是本申请实施例提供的一种计算设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例进行描述。
首先，对本申请中的部分用语进行解释说明，以便于本领域技术人员理解。
(1)增强现实(augmented reality，AR)，也被称之为混合现实。它通过电脑技术，将虚拟的信息应用到真实世界，真实的环境和虚拟的物体实时地叠加到了同一个画面或空间同时存在。增强现实提供了在一般情况下，不同于人类可以感知的信息。它不仅展现了真实世界的信息，而且将虚拟的信息同时显示出来，两种信息相互补充、叠加。
(2)抬头显示器(head up display，HUD)，也即平视显示器，是一种将图像投射到驾驶员前方视野中的显示装置。HUD最早应用于军用飞机，旨在降低飞行员需要低头查看仪表的频率。起初，HUD可以通过光学原理，将驾驶相关的信息投射在飞行员的头盔上，使得飞行员能在保证正常驾驶的同时关注到飞行的各项指标并接收地面传输的信息等，从而提升驾驶的安全性和便捷性。而今，HUD开始应用于汽车，设计初期，应用HUD的驱动力是保证驾驶的安全性，使得驾驶员在驾驶过程中无需低头将视线转移到仪表盘或者中控上，避免了人眼焦距在仪表盘或者中控和路面之间来回的切换，减少了危机反应的时间。因此，设计初期的HUD所投影的信息主要是汽车的行驶状况指标，比如仪表盘上所显示的车速、油量等较为简单的信息。
市面上常见的HUD可以包括投影仪、反射镜(或者称之为次反射镜)和投影镜(或者称之为主反射镜)等组件。HUD的成像原理类似于幻灯片投影，通过将图像投射到汽车挡风玻璃上，使得驾驶员可以获取视野前方的图像。具体地，一般可以先通过投影仪发出光信息，经过反射镜反射到投影镜上，再由投影镜反射到挡风玻璃，人眼看到的是位于眼前大约2-2.5米处的虚像，给人的感觉就是信息悬浮在前方路上。
需要说明的是，HUD图像投射在挡风玻璃上位置是可调的，通常情况下可以通过改变投影镜的角度调节HUD图像的位置。此外，可以理解的是，由于汽车的挡风玻璃是弯曲的，图像若是直接投射在弯曲的玻璃面上，会造成图像变形。这就需要一个纠正措施，因此，投影镜和反射镜往往会被设计成弯曲的。
如上所述，HUD利用光学反射原理，可以在挡风玻璃上投射显示超速预警、车况监控、油耗、时速等信息，能够让驾驶员的注意力集中在前方路面，实现主动的行车安全。同时还可以降低眼睛焦距需要不断调整产生的延迟与不适。
进一步地，设计人员希望通过HUD实现智能驾驶的目标，赋予其更多的功能。基于此，近年来提出了增强现实抬头显示器(augmented reality head up display，AR-HUD)，AR-HUD对于用户而言具有更强的直观性，其可以将HUD投射的图像与真实路面信息融合起来，增强驾驶员对路面信息的获取，例如可以实时投射显示一些虚拟箭头来直观地引导我们前进，从而避免在驾驶中出现开过路口和分散驾驶员注意力的情况。
请参阅图1，图1是一种AR-HUD成像效果示意图。如图1所示，AR-HUD投射到汽车挡风玻璃上所呈的虚像面可以位于驾驶员视野正前方。如图1所示，AR-HUD相较于以往的HUD而言，除了可以投射显示基本的驾驶时速和汽车电量等信息外，还可以显示AR导航指引箭头等信息，从而辅助驾驶员实现更加智能、舒适和安全的驾驶。如上所述，AR-HUD可以通过HUD投射的图像实现AR导航和AR预警等功能。可选地，具体可以实现例如跟车距离预警、压线预警、红绿灯监测、提前变道指示、行人预警、路标显示、车道偏离指示、前方障碍物预警、驾驶员状态监测等功能，此处不再进行赘述。
为了解决当前AR功能仿真以及测试技术中不满足实际业务需求的问题，本申请实施例提供了一系列方案，用于基于现有的计算设备(比如手机、车机、服务器等)，通过软件的方式构建各类场景，从而在该各类场景中对AR功能进行仿真，包括生成与上述各类场景对应的AR图像，并将各类场景与对应的AR图像进行融合，从而快速、高效地得到包括现实信息和虚拟信息的大量增强现实场景，以便捷、直观地体验各个场景下的AR效果。进一步，本申请实施例提供的一系列方案还可以根据仿真得到的增强现实场景，对AR功能进行测试，以不断改进、优化AR功能，提升AR效果。更进一步地，为了验证实际驾驶情况下可能存在的AR设备(例如上述AR-HUD)的安装偏差、人眼抖动和车辆抖动给AR成像效果带来的影响，本申请实施例提供的一系列方案还可以利用上述软件的方式，通过添加不同的畸变因素以及抖动量，仿真测试在各类非理想情况下的AR成像效果，以开发和不断改进相应的去畸变以及防抖动功能，提升AR效果，保证用户的使用体验。
请参阅图2，图2是本申请实施例提供的一种智能车辆的功能框图。本申请实施例提供的一种场景处理方法可以应用于如图2所示的智能车辆200中，在一个实施例中，智能车辆200可以配置为完全或部分地自动驾驶模式。在智能车辆200处于自动驾驶模式中时，可以将智能车辆200置为在没有和人交互的情况下操作。
智能车辆200可以包括各种子系统，例如行进系统202、传感系统204、控制系统206、一个或多个外围设备208以及电源210、计算机系统212和用户接口216。可选地，智能车辆200可包括更多或更少的子系统，并且每个子系统可包括多个元件。另外，智能车辆200的每个子系统和元件可以通过有线或者无线互连。
行进系统202可包括为智能车辆200提供动力运动的组件。在一个实施例中，行进系统202可包括引擎218、能量源219、传动装置220和车轮221。引擎218可以是内燃引擎、电动机、空气压缩引擎或者其他类型的引擎组合，例如汽油发动机和电动机组成的混动引擎，内燃引擎和空气压缩引擎组成的混动引擎。引擎218可以将能量源219转换成机械能量。
能量源219的示例包括汽油、柴油、其他基于石油的燃料、丙烷、其他基于压缩气体的燃料、乙醇、太阳能电池板、电池和其他电力来源。能量源219也可以为智能车辆200的其他系统提供能量。
传动装置220可以将来自引擎218的机械动力传送到车轮221。传动装置220可包括变速箱、差速器和驱动轴。在一个实施例中，传动装置220还可以包括其他器件，比如离合器。其中，驱动轴可包括可耦合到一个或多个车轮221的一个或多个轴。
传感系统204可包括若干个传感器，该若干个传感器可以用于采集关于智能车辆200周边的环境信息(例如可以包括智能车辆200周围的地形、道路、机动车辆、非机动车辆、行人、路障、交通标志、交通信号灯、动物、建筑和植物等等)。如图2所示，传感系统204可以包括定位系统222(定位系统可以是全球定位系统(global positioning system，GPS)系统，也可以是北斗系统或者其他定位系统)、惯性测量单元(inertial measurement unit，IMU)224、雷达226、激光测距仪228、相机230以及计算机视觉系统232等等。可选地，在一些可能的实时方式中，测试人员可以驾驶该智能车辆200至各类行车环境中(例如不同地区、不同地形、不同路况和不同天气的行车环境)，通过传感系统204内的多个传感器采集周围环境的数据，进一步地，还可以将采集到的数据上传至服务端。后续，测试人员可以从服务端获取 各个环境下采集到大量传感器数据，并基于该大量传感器数据通过计算设备构建对应的大量场景。可选地，测试人员也可以直接将采集到的大量数据发送至计算设备，等等，本申请实施例对此不作具体限定。其中，该大量场景可以作为AR功能测试中所需的现实信息，从而在进一步仿真得到虚拟HUD设备后，可以对HUD中涉及的AR功能等在各类场景中进行测试，观察各类场景下的AR成像效果，以不断优化相关AR功能，优化HUD的AR功能等，保证用户的使用体验。可选地，该智能车辆200中还可以包括AR-HUD(也即具备AR功能的HUD，图2中未示出)，可以通过将AR图像投射至驾驶员视野前方，以在驾驶过程中进行安全舒适的AR导航和AR预警等等，其中的AR-HUD可以为经上述仿真测试和改进后的AR-HUD。
其中，上述服务端可以为一个服务器，也可以为由多个服务器构成的服务器集群，或者还可以是一个云计算服务中心，等等，本申请实施例对此不作具体限定。其中，上述计算设备可以为智能可穿戴设备、智能手机、平板电脑、笔记本电脑、台式电脑或者带显示屏的服务器，等等，本申请实施例对此不作具体限定。
定位系统222可用于估计智能车辆200的地理位置。IMU 224用于基于惯性加速度来感测智能车辆200的位置和朝向变化。在一个实施例中，IMU 224可以是加速度计和陀螺仪的组合。
雷达226可利用无线电信号来感测智能车辆200的周边环境内的物体。在一些可能的实施例中，雷达226还可以用于感测智能车辆200周边车辆的速度和/或行进方向等等。其中，该雷达226可以为激光雷达，还可以为毫米波雷达，等等，可以用于采集周围环境的点云数据，继而可以得到点云形式的大量场景，用于对HUD进行仿真测试。
激光测距仪228可利用激光来感测智能车辆200所位于的环境中的物体。在一些可能的实施例中，激光测距仪228可包括一个或多个激光源、一个或多个激光扫描器以及一个或多个检测器，以及其他系统组件。
相机230可用于捕捉智能车辆200的周边环境的多个图像，继而可以得到照片形式的大量场景，用于对HUD进行仿真测试。在一些可能的实施例中，相机230可以是静态相机或者视频相机。
计算机视觉系统232可以操作来处理和分析由相机230捕捉的图像以便识别智能车辆200周边环境中的物体和/或特征。所述物体和/或特征可包括地形、机动车辆、非机动车辆、行人、建筑、交通信号、道路边界和障碍物等等。计算机视觉系统232可使用物体识别算法、运动中恢复结构(structure from motion，SFM)算法、视频跟踪和其他计算机视觉技术。
控制系统206为控制智能车辆200及其组件的操作。控制系统206可包括各种元件，其中包括油门234、制动单元236和转向系统240。
油门234用于控制引擎218的操作速度并进而控制智能车辆200的速度。
制动单元236用于控制智能车辆200减速。制动单元236可使用摩擦力来减慢车轮221。在其他实施例中，制动单元236可将车轮221的动能转换为电流。制动单元236也可采取其他形式来减慢车轮221转速从而控制智能车辆200的速度。
转向系统240可操作来调整智能车辆200的前进方向。
当然，在一个实例中，控制系统206可以增加或替换地包括除了所示出和描述的那些以外的组件。或者也可以减少一部分上述示出的组件。
智能车辆200通过外围设备208与外部传感器、其他车辆、其他计算机系统或用户之间 进行交互。外围设备208可包括无线通信系统246、车载电脑248、麦克风250和/或扬声器252。在一些实施例中，也可以通过无线通信系统246将传感器系统204中的一个或多个传感器采集到的数据上传至服务端，还可以通过无线通信系统246将传感器系统204中的一个或多个传感器采集到的数据发送至用于对HUD进行仿真测试的计算设备，等等，本申请实施例对此不作具体限定。
在一些实施例中，外围设备208提供智能车辆200的用户与用户接口216交互的手段。例如，车载电脑248可向智能车辆200的用户提供信息。用户接口216还可操作车载电脑248来接收用户的输入。车载电脑248可以通过触摸屏进行操作。在其他情况中，外围设备208可提供用于智能车辆200与位于车内的其它设备通信的手段。例如，麦克风250可从智能车辆200的用户接收音频(例如，语音命令或其他音频输入)。类似地，扬声器252可向智能车辆200的用户输出音频。
无线通信系统246可以直接地或者经由通信网络来与一个或多个设备无线通信。例如，无线通信系统246可使用第三代移动通信网络(3rd generation mobile networks，3G)蜂窝通信，例如码分多址(code division multiple access，CDMA)、全球移动通讯系统(global system for mobile communications，GSM)/通用分组无线业务(general packet radio service，GPRS)，或者第四代移动通信网络(4th generation mobile networks，4G)蜂窝通信，例如长期演进技术(long term evolution，LTE)。或者第三代移动通信网络(5th generation mobile networks，5G)蜂窝通信。无线通信系统246还可以利用无线保真技术(wireless-fidelity，WIFI)与无线局域网(wireless local area network，WLAN)通信。在一些实施例中，无线通信系统246可利用红外链路、蓝牙等与设备直接通信。其他无线协议，例如：各种车辆通信系统，例如，无线通信系统246可包括一个或多个专用短程通信(dedicated short range communications，DSRC)设备，这些设备可包括车辆和/或路边台站之间的公共和/或私有数据通信。
电源210可向智能车辆200的各种组件提供电力。在一个实施例中，电源210可以为可再充电锂离子或铅酸电池。这种电池的一个或多个电池组可被配置为电源为智能车辆200的各种组件提供电力。在一些实施例中，电源210和能量源219可一起实现，例如一些全电动车中那样。
智能车辆200的部分或所有功能受计算机系统212控制。计算机系统212可包括至少一个处理器213，处理器213执行存储在例如存储器214这样的非暂态计算机可读介质中的指令215。计算机系统212还可以是采用分布式方式控制智能车辆200的个体组件或子系统的多个计算设备。
处理器213可以是任何常规的处理器，诸如商业可获得的中央处理器(central processing unit，CPU)。可选地，该处理器可以是诸如特定应用集成电路(application-specific integrated circuit，ASIC)或其它基于硬件的处理器的专用设备。尽管图2功能性地图示了处理器、存储器和在相同块中的计算机系统212的其它元件，但是本领域的普通技术人员应该理解该处理器或存储器实际上可以包括不存储在相同的物理外壳内的多个处理器或存储器。例如，存储器可以是硬盘驱动器或位于不同于计算机系统212的外壳内的其它存储介质。因此，对处理器或存储器的引用将被理解为包括对可以或者可以不并行操作的处理器或存储器的集合的引用。不同于使用单一的处理器来执行此处所描述的步骤，例如传感系统204中的一些组件每个都可以具有其自己的处理器，所述处理器只执行与特定于组件的功能相关的计算。
在此处所描述的各个方面中，处理器213可以位于远离该车辆并且与该车辆进行无线通 信。在其它方面中，此处所描述的过程中的一些在布置于车辆内的处理器上执行而其它则由远程处理器执行。
在一些实施例中，存储器214可包含指令215(例如，程序逻辑)，指令215可被处理器213执行来执行智能车辆200的各种功能，包括以上描述的那些功能。存储器214也可包含额外的指令，包括向行进系统202、传感系统204、控制系统206和外围设备208中的一个或多个发送数据、从其接收数据、与其交互和/或对其进行控制的指令。
除了指令215以外，存储器214还可存储数据，例如传感器系统204在行驶过程中采集到的大量传感器数据，比如可以包括传感器系统204内的相机230拍摄到的图像数据以及雷达226采集到的点云数据，等等。在一些实施例中，存储器214还可存储例如道路地图、路线信息，车辆的位置、方向、速度以及其它这样的车辆数据，以及其他信息，等等。这种信息可在智能车辆200行驶期间被智能车辆200中的无线通信系统246或者计算机系统212等使用。
用户接口216，用于向智能车辆200的用户提供信息或从其接收信息。可选地，用户接口216可包括在外围设备208的集合内的一个或多个输入/输出设备，例如无线通信系统246、车车在电脑248、麦克风250和扬声器252。
可选地，上述这些组件中的一个或多个可与智能车辆200分开安装或关联。例如，存储器214可以部分或完全地与智能车辆200分开存在。上述组件可以按有线和/或无线方式来通信地耦合在一起。
综上所述，智能车辆200可以为轿车、卡车、摩托车、公共汽车、船、无人机、飞机、直升飞机、割草机、娱乐车、游乐场车辆、施工设备、电车、高尔夫球车、火车、和手推车，等等，本申请实施例对此不作具体限定。
可以理解的是，图2中的智能车辆的功能框图只是本申请实施例中的一种示例性的实施方式，本申请实施例中的智能车辆包括但不仅限于以上结构。
请参阅图3a，图3a是本申请实施例提供的一种场景处理方法的系统架构示意图，本申请实施例的技术方案可以在图3a举例所示的系统架构或者类似的系统架构中具体实施。如图3a所示，该系统架构可以包括计算设备100和多个智能车辆，具体可以包括智能车辆200a、200b和200c。其中，计算设备100、智能车辆200a、200b和200c之间可以通过有线或者无线网络(例如无线保真(Wireless-Fidelity，WiFi)、蓝牙和移动网络)等方式互相建立连接。其中，智能车辆200a、200b和200c中均可以安装有多个传感器，例如相机、激光雷达、毫米波雷达、温度传感器和湿度传感器等等。在行驶过程中，智能车辆200a、200b和200c可以通过车内的多个传感器采集周围环境的数据(可以包括图像、点云、温度和湿度等数据)，并通过有线或者无线的方式将数据发送至计算设备100。计算设备100可以基于该大量传感器数据构建大量仿真场景，生成与各个场景相对应的AR图像并与其进行融合，以获得相应的增强现实场景，最终显示得到的增强现实场景。
可选地，请参阅图3b，图3b是本申请实施例提供的另一种场景处理方法的系统架构示意图，本申请实施例的技术方案可以在图3b举例所示的系统架构或者类似的系统架构中具体实施。如图3b所示，该系统架构除上述计算设备100以及智能车辆200a、200b和200c外，还可以包括服务端300。相应的，智能车辆200a、200b和200c在行驶过程中采集到的数据均可以通过网络上传至服务端300中，后续计算设备100可以通过网络从该服务器中获取相应 的传感器数据，等等，此处不再进行赘述。如此，可以更加便捷地实现传感器数据的共享，使得不同的测试人员均可以根据实际需求通过网络从服务端获取智能车辆200a、200b和200c采集到的数据，提高测试效率。
如上所述，计算设备100在获取到智能车辆200a、200b和200c内的多个传感器采集到的数据后，可以基于该大量传感器数据构建大量仿真场景，生成与各个场景相对应的AR图像并与其进行融合，以获得相应的增强现实场景(也即各类场景下的AR效果)，最终显示得到的增强现实场景。至此，计算设备100完成了通过软件仿真的方法，不依赖于HUD硬件设备，高效、直观地得到了各类不同的场景下的AR效果，大大降低了AR功能展示的成本，并且，后续可以根据该各类不同的场景下的AR效果对HUD的AR功能进行全面、高效地测试，以不断优化AR功能，保证用户的使用体验。并且，极大程度上节省了测试人员外出进行实车测试所需消耗的时间资源和人力物力，大大降低了测试成本。可选地，测试人员可以驾驶智能车辆200a、200b和200c至各类地形、路况和天气等行驶环境中，以采集各类场景的数据，从而可以为后续HUD的AR功能仿真测试，以及抖动或者畸变因素影响下的AR成像效果仿真测试提供大量、可复现场景的支撑，大大提高测试结果的可靠性。
或者，如上所述，服务端300接收智能车辆200a、200b和200c上传的传感器数据，该数据可以为传感器采集到的原始数据，也可以是经过传感器预处理(例如筛选、融合等)后的数据，等等。服务端300可以基于该大量传感器数据通过本申请实施例中的方法构建大量仿真场景，生成与各个场景相对应的AR图像并与其进行融合，以获得相应的增强现实场景。随后，服务端300将获得的增强显示场景通过消息发送至计算设备100，使得计算设备100通过显示屏显示该增强现实场景。可选地，该计算设备100还可以为智能车辆200a、200b和200c内的显示屏等。
此外，在一些特殊场景中，例如实时直播场景，主播在驾驶车辆200a行驶过程中，车载传感器采集到的数据数据(还可以包括车上的直播设备(图中未示出)采集到的数据，例如主播的手机终端采集到的周围环境的视频数据等)可以实时上传至服务端300。如此，服务端300可以接收实时采集到的数据，并基于此，实时构建仿真场景，还原主播周围的环境，以及实时生成与其对应的AR图像并与其进行融合，以获得相应的增强现实场景，并将获得的增强现实场景发送给直播设备，以使得直播设备的显示屏显示该增强现实场景。然后，观众便可以利用手中的手机或者平板等通过网路直播间观看实时的增强现实场景(例如包括主播拍摄的旅游场景以及服务器融合的天气、景点标识和景点介绍等AR图标等)。如上所述，服务器300进行仿真模拟所使用的场景数据可以是事先采集到的数据，也可以是实时数据，本申请实施例对此不作具体限定。
可选地，请参阅图3c，图3c是本申请实施例提供的一种计算设备的功能框图。如图3c所示，该计算设备100可以包括人眼姿态模拟模块101、HUD参数模拟模块102、传感器数据模块103、数据融合模块104、AR功能模块105、渲染引擎模块106、HUD虚拟模块107、场景生成模块108和显示模块109。
其中，人眼姿态模拟模块101，可以用于模拟驾驶员在驾驶过程中的人眼姿态，包括车辆颠簸时驾驶员人眼位置的抖动变化等，可以理解的是，人眼姿态的变化可以影响驾驶员观测到的HUD投射出的AR成像效果。
其中，HUD参数模拟模块102，可以用于设置HUD相关的硬件参数。应理解，如上所 述，HUD自身硬件或不同挡风玻璃曲率均容易造成AR图像的畸变，因此，可以通过HUD参数模拟模块102参数化模拟各项硬件参数，从而为后续的仿真测试提供大量、全面的输入参数。例如可以包括挡风玻璃曲率、眼盒位置、HUD虚像面尺寸和HUD安装位置等参数，本申请实施例对此不作具体限定。
其中，传感器数据模块103，可以用于获取以及存储多个传感器采集到的大量数据(例如上述智能车辆200a、200b和200c内的多个传感器在行驶过程中采集到的数据)。可选地，该传感器数据还可以为通过现有的仿真模拟系统(或者说仿真软件)构建出的虚拟传感器中设置的数据，等等。由此，相应地，在一些可能的实施例中，上述系统架构也可以仅包括计算设备，本申请实施例对此不作具体限定。从而无需测试人员在车辆内装配传感器，并驾驶车辆至实际道路中进行数据采集，进一步节省了人力、物力和时间资源，大大提升了测试效率，降低了测试成本，并且，虚拟传感器不依赖于实车和真实场景，通过简单的数据设置以及变化，可以更加全面、高效地构建出各类不同的场景。可以理解的是，测试人员可以根据实际条件和测试需求，选择实车采集数据或者设置虚拟传感器数据，等等，本申请实施例对此不作具体限定。
其中，数据融合模块104，可以用于将传感器模块103中的数据进行融合，例如将相机拍摄到的图像数据与激光雷达采集到的点云数据进行融合等等，从而为后续构建测试所需的场景提供更加全面、有效的数据支撑，提高场景的质量。可选地，如图3c所示，数据融合模块104还可以接收人眼姿态模拟模块101和HUD参数模拟模块102中的相关数据，进而可以在后续虚拟HUD设备投射出的虚像面中直观的显示各种硬件参数变化和人眼姿态变化对AR成像效果的影响。
其中，AR功能模块105，可以包括相应的一系列模型和软件算法等等，可以用于生成与各类场景等相匹配的AR图像，每个AR图像中可以包括一个或多个AR图标，例如AR导航指引箭头、驾驶时速和汽车电量等等。
其中，渲染引擎模块106，可以用于将相应的一个或多个AR图标渲染至HUD投射出的虚像面中。
其中，HUD虚拟模块107，可以用于基于上述HUD参数模拟模块102中的相关参数对HUD进行仿真模拟，构建相应的HUD虚像面，等等。
其中，场景生成模块108，可以用于基于上述传感器数据模块103中的传感器数据直接构建大量场景，还可以用于基于上述数据融合模块104中对各类传感器数据进行融合后得到的数据构建大量场景，等等。进一步地，还可以通过上述HUD虚拟模块107在该大量场景中构建相应的HUD虚像面。
其中，显示模块109，可以用于在当前场景的HUD虚像面中显示基于当前场景、模型以及前述相应的硬件参数和人眼姿态生成的AR图像，以便测试人员可以直观的掌握当前情况下的AR成像效果，进一步根据该AR成像效果分析模型或算法可能存在的问题，以优化AR功能。以及还可以根据该AR成像效果分析各种硬件参数和人眼姿态变化对AR成像效果的影响，从而为后续的去畸变功能、防抖动功能的开发和测试提供有效支撑，不断提升用户的使用体验。
可以理解的是，图3c中的计算设备的功能框图只是本申请实施例中的一种示例性的实施方式，本申请实施例中的计算设备包括但不仅限于以上结构。
综上所述，计算设备100可以是具备上述功能的智能手机、智能可穿戴设备、平板电脑、 笔记本电脑、台式电脑和车机,等等，本申请实施例对此不作具体限定。智能车辆200a、200b和200c可以是具备上述功能的家用小轿车、面包车、公交车、出租车、摩托车和游艇等等，本申请实施例对此不作具体限定。服务端300可以是具备上述功能的计算机和服务器等等，服务端300可以是一台服务器，也可以是由多台服务器组成的服务器集群，或者是一个云计算服务中心，服务端300可以为计算设备100和智能车辆200a、200b和200c提供后台服务，例如为车联网服务平台，等等，本申请实施例对此不作具体限定。
为了便于理解本申请实施例，以下示例性列举本申请中的一种场景处理方法所适用的应用场景，可以包括如下场景。
场景一，基于车载传感器针对真实场景采集到的数据，构建场景，对HUD进行仿真测试。
请参阅图4，图4是本申请实施例提供的一种场景处理方法的应用场景示意图。如图4所示，该应用场景可以包括计算设备100以及行驶在实际道路(例如图4所示的现实世界中较为常见的多车道公路)中的智能车辆200(图4中以小轿车为例)。可选地，如图4所示，该应用场景中还可以包括多部其他车辆，例如车辆1(图4中以公交车为例)、车辆2(图4中以小轿车为例)和车辆3(图4中以小轿车为例)。其中，计算设备100与智能车辆200可以通过有线或者无线的方式建立通信连接，智能车辆200可以为上述智能车辆200a、200b和200c中的任意一个，内置传感器系统，包括多个传感器(例如相机、激光雷达和毫米波雷达等等)。智能车辆200在该道路上行驶时可以通过车辆的多个传感器采集周围环境的数据，并将数据发送至计算机设备100。然后，如图4所示，计算设备100可以通过本申请实施例提供的一种场景处理方法，基于智能车辆200内的多个传感器采集到的数据，构建大量场景。显然，由于为实车驾驶到实际道路中进行数据采集，则该大量场景可以均为实景仿真场景。进一步地，计算设备100可以通过软件仿真的方式，基于预设的HUD硬件参数，构建虚拟HUD，从而在该多个实景仿真场景中生成对应的AR图像并进行融合，以高效、便捷地获得大量增强现实场景并进行显示，如此，无需用户驾驶实车去道路中，便能直观的体验到各类场景下HUD产品的AR功能，为用户提供了便利。并且，还可以基于该大量增强现实场景对HUD的AR功能等进行测试，如上所述，本申请实施例可以不依赖于HUD硬件设备，在各类不同的场景下对HUD的AR功能进行全面、高效地测试，以不断优化AR功能，保证用户的使用体验，并且大大降低了测试成本。
如上所述，计算设备100可以是具备上述功能的智能手机、智能可穿戴设备、平板电脑、笔记本电脑、台式电脑、车机等等，本申请实施例对此不作具体限定。智能车辆200可以是具备上述功能的家用小轿车、面包车、公交车、出租车、摩托车和游艇等等，本申请实施例对此不作具体限定。
场景二，基于虚拟传感器中设置的数据，构建场景，对HUD进行仿真测试。
请参阅图5，图5是本申请实施例提供的另一种场景处理方法的应用场景示意图。如图5所示，该应用场景可以包括计算设备100。如图5所示，计算设备100可以首先通过现有的仿真模拟系统(或者说仿真软件)构建多个虚拟传感器(例如可以包括虚拟相机、虚拟激光雷达和虚拟毫米波雷达等等)，然后对每个虚拟传感器的数据进行设置，例如设置虚拟相机中的虚拟物体，比如虚拟车辆、虚拟道路等等。然后，计算设备100可以基于该多个虚拟传感 器的数据，构建多个场景，显然，相应的，该多个场景可以均为虚拟仿真场景。如图5所示，该虚拟仿真场景可以类似于第一人称视角下的游戏场景，等等，本申请实施例对此不作具体限定。进一步地，计算设备100可以通过软件仿真的方式，基于预设的HUD硬件参数，构建虚拟HUD，从而在该多个实景仿真场景中生成对应的AR图像并进行融合，以高效、便捷地获得大量增强现实场景并进行显示等等，此处不再进行赘述。进一步地，相较于图4所示的需实车驾驶进行传感器数据采集的方案而言，图5所示的应用场景进一步节省了工作人员外出进行数据采集所需消耗的时间资源和人力物力等，进一步降低了AR功能展示以及AR功能测试的成本，提高了测试效率。
如上所述，计算设备100可以是具备上述功能的智能手机、智能可穿戴设备、平板电脑、笔记本电脑、台式电脑、车机等等，本申请实施例对此不作具体限定。
场景三，基于实时采集的数据，进行直播。
请一并参阅图4，在一些实时直播场景中，例如AR游戏直播或者相关产品的推广体验直播活动等场景，主播在驾驶车辆200行驶过程中，车载传感器采集到的数据(还可以包括直播设备(图中未示出)采集到的数据，例如主播的手机终端采集到的周围环境的视频数据等)可以实时上传计算设备100。如此，计算设备100可以接收实时采集到的数据，并基于此，实时构建仿真场景，还原主播周围的环境，以及实时生成与其对应的AR图像并与其进行融合，以获得相应的增强现实场景，并将获得的增强现实场景发送给直播设备，以使得直播设备的显示屏显示该增强现实场景。然后，观众便可以利用手中的手机或者平板等通过网路直播间观看实时的增强现实场景(例如包括主播拍摄的旅游场景以及服务器融合的天气、景点标识和景点介绍等AR图标等)。其中，计算设备100可以具备上述功能的服务器，可以是一台服务器，也可以是由多台服务器构成的服务器集群，或者是一个云计算服务中心，等等，本申请实施例对此不作具体限定。
可以理解的是，上述应用场景只是本申请实施例中的几种示例性的实施方式，本申请实施例中的应用场景包括但不仅限于以上应用场景，其它场景及举例将不再一一列举和赘述。
请参阅图6a，图6a是本申请实施例提供的一种场景处理方法的流程示意图。该方法可应用于图3a和图3b所述的系统架构以及上述图4和图5所述的应用场景中，其中的计算设备可以用于支持并执行图6a中所示的方法流程步骤S801-步骤S804。下面将结合附图6a从计算设备侧进行描述，该方法可以包括以下步骤S801-步骤S804。
步骤S801，获取第一车外场景；其中，所述第一车外场景为二维的场景图像或者三维的场景模型。
具体的，计算设备该第一车外场景可以为二维的场景图像，也可以为三维的场景模型，等等，本申请实施例对此不作具体限定。其中，该场景元素例如可以包括车道、车辆、植物、行人、动物和交通信号，等等，本申请实施例对此不作具体限定。
步骤S802，获取与所述第一车外场景对应的第一AR图像。
具体地，计算设备可以基于第一车外场景以及预设模型，对第一车外场景进行识别，获取与该第一车外场景对应的第一AR图像。
可选地，每个第一AR图像可以包括一个或多个AR图标(例如为注意前方车辆和人行道等的警示图标或者文字信息，又例如相关的景点介绍和天气信息等，以及还可以包括相关 AR导航标识，比如直行箭头和左/右转弯箭头等等，本申请实施例对此不作具体限定)。可选地，每个第一车外场景和每个AR图标均可以有对应的一个或多个属性，该属性比如天气、季节、地理位置、道路状况、地形、交通信号等等。其中，该第一AR图像中包括的每个AR图标的属性可以为与该第一车外场景属性相同(或者说相匹配)。例如，该第一车外场景的属性包括学校、道路驾驶，则可以基于该第一车外场景以及预设模型，获取相对应的学校标识和减速慢行等AR图标。
步骤S803，融合所述第一车外场景和所述第一AR图像，以获得第二场景。
具体地，计算设备可以第一车外场景和其对应的第一AR图像进行融合，以获得对应的第二场景。其中，第一车外场景为第二场景中的现实信息，第一AR图像为第二场景中的虚拟信息。
步骤S804，使能显示屏显示所述第二场景。
具体地，该显示屏可以为该计算设备的显示屏，则计算设备在获得该第二场景后可以通过其显示器进行显示。可选地，该显示屏还可以是其他设备的显示屏，则计算设备在获得该第二场景后，可以将该第二场景发送至其他设备，以使得该其他设备通过其显示屏显示该第二场景，等等，本申请实施例对此不作具体限定。可选地，还可以参考上述图3b和场景三对应实施例的描述，此处不再赘述。
如此，本申请实施例可以基于软件的方法，通过现有的计算设备(比如笔记本电脑和台式电脑等)构建大量二维、或三维的仿真场景；然后，基于上述仿真场景以及开发的模型，生成与上述仿真场景对应的AR图像，并将上述仿真场景与对应的AR图像进行融合，从而快速、高效得到大量包括现实信息和虚拟信息的增强现实场景。
进一步地，需要说明的是，为了验证AR-HUD中涉及的AR地图、AR导航和兴趣点(point of interest，POI)等一系列软件算法功能在真实路况下的效果，保证用户在各类场景下行驶时均可以获得正确的AR导航或者AR预警等信息，往往需要对上述预设模型以及一系列软件算法等进行多次测试以及优化。而现有技术中通常需要将装有AR-HUD设备的汽车驾驶到现实世界中进行多路况、多天气、多地区以及多场景的重复功能测试，也即进行实车测试。而在实车测试过程中，若发现AR功能失效，例如对于需要直行的道路，出现了右转弯或者掉头等错误的AR导航指引箭头，导致出现该问题的原因究竟是汽车内各个传感器的数据不同步、AR事件响应逻辑错误、还是硬件设备故障等，后续都需要在测试结束后对出现的问题进行线下的分析测定，过程复杂繁琐，且实车测试需要耗费大量的时间、人力和物力，成本较高。另外，在对AR功能改进之后，还需要对AR功能再次进行充分测试已验证前述问题得以解决，但是，由于实际的道路状况、天气等等始终在变化，因此若想要复现之前的失效场景，以对AR功能再次进行测试则十分困难，如此极大程度上降低了测试结果的可靠性以及测试效率。相较于现有技术，本申请实施例可以不依赖于现实场景以及HUD硬件设备，不仅可以快速、高效地展示各类增强现实场景，后续还可以基于本申请实施例仿真得到的各类增强现实场景下直观的AR效果，不断优化和改进HUD的AR功能，保证用户的使用体验。
可以理解的是，不同的方法或者步骤可以由不同的执行主体来执行，下面，以计算设备100为例对本申请实施例提供的另一种场景处理方法展开介绍。请参阅图6b，图6b是本申请实施例提供的另一种场景处理方法的流程示意图。该方法可应用于图3a和图3b所述的系统架构以及上述图4和图5所述的应用场景中，其中的计算设备可以用于支持并执行图6b中所 示的方法流程步骤S901-步骤S903。下面将结合附图6b从计算设备侧进行描述，该方法可以包括以下步骤S901-步骤S903。
步骤S901，获取目标场景集合，所述目标场景集合包括X个第一车外场景；每个第一车外场景包括一个或多个场景元素。
具体地，计算设备获取目标场景集合，该目标场景集合中可以包括X个第一车外场景，每一个第一车外场景中可以包括一个或多个场景元素。可选地，步骤S901可参考上述图6a对应实施例中的步骤S801，此处不再进行赘述。其中，X为大于或者等于1的整数。
可选地，计算设备可以获取N个第一传感器采集的数据，该N个第一传感器可以为设置在目标车辆(例如为上述智能车辆200a、200b和200c中的任意一个)上的车载传感器(例如为相机、激光雷达和毫米波雷达等)。该N个第一传感器采集的数据可以为在该目标车辆行驶过程中针对该目标车辆的周围环境采集的数据。其中，N为大于或者等于1的整数。然后，计算设备可以基于该N个第一传感器采集的数据，并通过现有的渲染引擎(例如开放式图形库(Open Graphics Library，openGL)、unreal(虚幻引擎)和unity(一种渲染引擎)等)构建对应的K个第一车外场景。可以理解的是，该K个第一车外场景可以为实景仿真场景。其中，K为大于或者等于1，且小于或者等于X的整数。
可选地，计算设备还可以获取M个第二传感器采集的数据，该M个第二传感器可以为通过预设的仿真模拟系统构建的虚拟传感器。该M个第二传感器采集的数据可以为通过上述预设的仿真模拟系统设置的数据。其中，M为大于或者等于1的整数。然后，相应的，计算设备可以基于该M个第二传感采集的数据，并通过现有的渲染引擎构建对应的P个第一车外场景。可以理解的是，相应的，该P个第一车外场景可以为虚拟仿真场景(类似于虚拟驾驶游戏场景)。P为大于或者等于1，且小于或者等于X的整数。
可选地，请参阅图7，图7是本申请实施例提供的另一种场景处理方法的流程示意图。步骤S801可以参考图7所示的方法流程。如图7中的步骤S11所示，首先，计算设备可以利用现有的仿真模拟系统初始化基本场景设置，例如可以包括交通、地图、天气、车辆和多种传感器等基本单元信息。然后，测试人员可以根据实际的测试条件和测试需求，选择通过实车驾驶采集到的传感器数据生成相应的实景仿真场景(例如上述的K个第一车外场景)，或者，可以选择通过设置虚拟传感器中的数据生成相应的虚拟仿真场景(例如上述的P个第一车外场景)。
例如，如图7中的步骤S12a和步骤S13a所示，如果选择不通过实车数据，则渲染引擎将通过类似于虚拟驾驶游戏场景的构建的方式，通过虚拟传感器的数据输入完成各个基本单元信息的实例化操作，也即整个场景将通过模拟仿真的方式生成。可选地，请参阅图8，图8是本申请实施例提供的一种场景的示意图。如图8所示，该第一车外场景为虚拟仿真场景(图8中以二维的场景图像为例)，其中可以包括车辆、交通信号灯(也即红绿灯)、人行道、多个植物和直行道等等场景元素，此处不再进行赘述。可选地，该第一车外场景中还可以包括驾驶员第一人称视角下的方向盘、汽车挡风玻璃和汽车引擎盖等等元素(图8中未示出)，本申请实施例对此不作具体限定。可选地，在一些可能的实施例中，也可以基于虚拟传感器的数据输入，并通过现有的渲染引擎构建得到三维立体的场景模型，继而后续可以在该三维立体的场景模型中进行HUD的AR功能仿真测试，以提高测试的可观性和精确性，等等，本申请实施例对此不作具体限定。
例如，如图7中的步骤S12a和步骤S13a所示，如果选择通过实车数据，则计算设备可 以通过渲染引擎读取相应传感器中的图像和点云等数据，利用SFM/同步定位与地图构建(Simultaneous Localization And Mapping，SLAM)算法的方式进行离线/在线的三维稀疏/稠密场景重构，同时用激光雷达、GPS/IMU等传感器的数据实例化基本单元，整个场景将通过实景合成方式生成，生成的场景可以通过object(对象文件)、mesh(网格)等存储形式导入至渲染引擎中。可选地，相应的，通过真实传感器采集到的数据生成的场景可以为上述场景，也可以为三维立体的场景模型，等等，此处不再进行赘述。
可选地，请参阅图9，图9是本申请实施例提供的一种相机坐标系和世界坐标系的示意图。如图9所示，场景中物体的空间三维(3 dimensions，3D)坐标{x，y，z}与真实相机传感器拍摄到的图像2D坐标{u，v}之间应满足关系：
Figure PCTCN2021084488-appb-000001
其中，f x和f y为焦距，c x和c y为O o距离中心O c的偏移量，K为相机内参。如此，建立了物体真实3D位置到以相机光心为中心的2D坐标的关系，真实地通过相机的内外参将相机拍摄数据和物体的实际坐标对应，随即构建了单帧场景。例如，请参阅图10，图10是本申请实施例提供的一种场景重构的示意图。如图10所示，通过单张相机图像或者单个场景的点云数据可以进行局部场景重构，得到单帧场景。此外，请一并参阅图10，通过多帧图像的重建技术，可以将传感器采集到的多模态数据(例如多张不同角度、不同场景的相机图像和多个激光雷达针对不同场景采集到的点云数据等)进行融合，进而重建出整个全局场景，等等，后续也可以在该全局场景对HUD的AR功能进行测试，本申请实施例对此不作具体限定。如此，通过上述步骤可以基于软件的方法生成大量各种各样的虚拟仿真场景和实景仿真场景，从而为HUD的AR成像效果测试提供了基础环境支持。
步骤S902，在所述X个第一车外场景中融合与所述X个第一车外场景对应的X个第一AR图像，生成对应的X个第二场景；每个第一AR图像包括一个或多个AR图标。
具体地，步骤S902可以参考上述图6a对应实施例中的步骤S802和步骤S803，此处不再进行赘述。
可选地，该预设模型可以为神经网络模型，该神经网络模型可以是根据由多个场景、多个AR图标以及多个场景和多个AR图标的不同匹配度训练得到的。对于该神经网络模型可以由一下示例性训练方法获得：以多个场景作为训练样本，在该多个场景中叠加与其匹配的一个或多个AR图标，得到对应的多个增强现实场景，以该多个增强现实场景为目标，通过深度学习算法来训练，以获得与目标接近的结果，并获得相应的神经网络模型。其中，上述训练样本中的场景可以是利用摄像头拍摄得到的图片，也可以是由激光雷达扫描得到的点云图像等，本申请实施例对此不作具体限定。
可选地，请参考图7所示的步骤S14，在场景构建完成后，首先，计算设备可以基于预设的HUD参数集合，对HUD进行仿真模拟，在该X个第一车外场景中构建对应的X个HUD虚像面，其中HUD虚像面的构建也可以使用上述现有的渲染引擎技术来实现。其中，该HUD参数集合可以包括挡风玻璃曲率、眼盒位置、人眼观测位置、HUD安装位置和HUD虚像面尺寸等等中的至少一种参数。例如，请参阅图11，图11是本申请实施例提供的一种HUD虚像面的示意图。图11中以第一车外场景为虚拟仿真场景为例，该HUD虚像面可以为该第一 车外场景中对应的一个区域，图11可以为驾驶员视角下的HUD仿真模拟效果，也即相当于在“驾驶员”的视野前方构建了一个尺寸大小和真实HUD硬件参数相匹配的“屏幕”，后续AR功能效果(也即AR图像)可以渲染绘制于该“屏幕”上，其中，渲染可以采用不限于离屏渲染的方式来实现。
需要说明的是，在理想状态下，即不考虑HUD畸变、人眼视线抖动，以及车辆颠簸造成的HUD空间位置发生变化的情况下，当前述的第一车外场景开始渲染时，驾驶员的人眼位置应在HUD标定好的眼盒位置当中，即人眼位置在已知的最佳观测位置，驾驶员在该位置观察虚像面上的物体(也即AR图像)是最清晰、最好的。如此，在前述第一车外场景构建完成后，可以在第一车外场景中构建一个距离为l、高度为h、宽度为w、透射率为alpha(阿尔法)的HUD虚像面，该HUD虚像面可以完全承担HUD的仿真模拟功能。除此之外，还可以在眼盒位置，利用相机模型来模拟真实人眼观测，前述的第一车外场景以及虚拟HUD的AR成像效果(例如第一AR图像)均可以在观测位置成像。
可选地，在HUD虚像面构建完成后，可以根据HUD虚像面尺寸，裁切相应的图像帧用于内存缓冲中，并将基于预设模型生成的与第一车外场景对应的第一AR图像渲染至该HUD虚像面中，从而生成对应的第二场景，如此，即可观察得到理想状态下HUD的AR成像效果。可选地，在将第一AR图像渲染至该HUD虚像面之前，还可以先对该第一AR图像进行一定的预处理，例如裁剪、缩放、旋转等处理，以使得该第一AR图像与该第一车外场景以及该HUD虚像面相适配。
例如，请参阅图12a，图12a是本申请实施例提供的一种AR成像效果的示意图。如图12a所示，该第二场景可以继承第一车外场景中的多个场景元素，此外，该第二场景包括HUD虚像面，以及显示于该HUD虚像面上的多个AR图标(例如可以包括如图12a所示的当前时速信息和当前电量信息，车辆、人行道和交通信号灯的警示信息，以及直行的AR导航指引箭头，等等)。
例如，请参阅图12b，图12b是本申请实施例提供的另一种AR成像效果的示意图。如图12b所示，显然，该第二场景为实景的场景，包括车道、车辆和植物等场景元素，还包括构建的HUD虚像面以及该HUD虚像面中的多个AR图标(图12b中以车辆的目标框为例，可以用于提示驾驶员注意车道旁的车辆，从而保证驾驶安全，等等)。
例如，请参阅图12c，图12c是本申请实施例提供的又一种AR成像效果的示意图。如图12b所示，显然，该第二场景可以为基于点云数据构建得到的场景，其中也可以包括多个场景元素、HUD虚像面以及AR图标等等。
可选地，步骤S902中涉及的具体计算公式可以如下：
其中，物体的空间位置在相机模型(也即人眼)中的投影v’可利用下述投影变换关系进行计算：
v'＝M proj·M view·M model·v
其中，
投影(projection)矩阵为：
Figure PCTCN2021084488-appb-000002
其中，near为近 平面，far为远平面，top为顶端。
视角(view)矩阵为：
Figure PCTCN2021084488-appb-000003
模型(model)矩阵为：M model＝identity 4
相机视场角(field of view，Fov)为：
Figure PCTCN2021084488-appb-000004
其中，L h为相机成像近平面高度，f y为相机焦距。
综上，经过上述投影模型以及下述的标准化设备坐标(Normalized Device Coordinates，NDC)转换关系，再通过采用帧缓冲技术，即可以将待渲染的AR图标绘制于准确的计算设备屏幕坐标上。
Figure PCTCN2021084488-appb-000005
可选地，请参阅图13，图13是本申请实施例提供的一种外部视角下的HUD模拟仿真效果示意图。如上所述，通过上述步骤，在不考虑任何HUD畸变、人眼位置变化和车辆抖动因素的影响下，可以完全模拟如图13所示的实际行车过程中的HUD成像过程。可选地，如图13所示，其中，上述相机视场角可分为垂直视场角和水平视场角，垂直视场角即为HUD虚像面顶部和底部所呈的角度，水平视场角即为HUD虚像面左边缘和右边缘所呈的角度，此处不再详述。
步骤S903，基于所述每个第二场景中的所述一个或多个场景元素以及所述一个或多个AR图标，对AR效果进行分析。
具体地，计算设备在生成第二场景后，可以基于该每个第二场景中的一个或多个场景元素以及一个或多个AR图标之间的位置关系和/或逻辑关系，对当前的AR效果进行分析，进一步地，可以对该预设模型以及一系列软件算法等进行相应的修正，以不断优化HUD的AR功能，保证用户的使用体验。可选地，在优化AR功能后，还可以基于与之前测试用的相同的第一车外场景再次进行重复测试，并对比前后的AR成像效果，已验证其是否得到有效的改进，等等，从而保证测试的严谨性和测试结果的可靠性。
请参阅图14，图14是本申请实施例提供的又一种AR成像效果的示意图。相较于图12a所示的AR导航指引箭头，图14所示的AR导航指引箭头的位置明显与场景元素(直行道)的位置之间存在偏差，其与路面不贴合。如此，测试人员可以直观的根据当前仿真得到的AR成像效果，快速定位当前算法可能存在的问题与不足，从而更准确、高效、有针对性地对算法进行改进。
应理解，HUD的成像效果受多种因素影响，一般来说，在将HUD设备装载于汽车时， 往往需要对HUD的失真畸变、虚像距离和眼盒位置等参数进行标定。在获得了准确的标定参数后，当驾驶员在驾驶舱内观察且驾驶员人眼位置位于眼盒范围内时，便可以观测到清晰的HUD虚像投影，从而获得极佳的AR视觉效果。然而，在HUD实车装配过程中，往往由于装配精度问题造成HUD的安装位置存在偏差，进而导致HUD的虚像距离(virtual image distance，VID)发生变化，造成HUD投射的AR图像(例如AR导航指引箭头等)产生畸变。另外，挡风玻璃的不同类型(例如不同的挡风玻璃曲率)也会带来非线性参数畸变，使得HUD投射的AR图像失真畸变，从而大大影响用户的视觉感受和使用体验。为此，市场上大多数投入生产制造的HUD往往会添加相应的去畸变功能，以应对上述可能存在的图像畸变情况，保证用户体验。然而，目前在去畸变功能的开发和测试过程中，大多采取实车测试的方法，也即需要在实车标定安装HUD后，才能够对由于不同安装位置以及风道玻璃曲率等因素造成的AR图像畸变以及AR渲染精度下降的具体影响进行直观地评测，以对HUD的去畸变功能进行不断的改进，保证用户的使用体验，其过程复杂、耗时长、效率低以及成本高。
基于此，进一步地，本申请实施例还可以基于上述软件仿真的方法，通过对HUD参数化的扩展支持，可以在场景中直接对渲染缓冲的内存值进行畸变变换(也即对待渲染的AR图像进行畸变处理)，实时渲染以直观的模拟由于参数变化对AR成像效果带来的影响。从而对HUD安装位置存在偏差以及挡风玻璃曲率不标准等畸变因素造成的AR渲染精度下降等具体影响进行直观地评测。
可选地，可以通过计算设备对Q个第一车外场景对应的Q个第一AR图像进行畸变处理，获取对应的Q个第二AR图像；融合该Q个第一车外场景和该Q个第一AR图像，以获得对应的Q个第三场景，并使能显示屏显示该Q个第三场景。相应的，每个第二AR图像也可以包括一个或多个AR图标。其中，该畸变处理可以包括径向畸变、切向畸变、虚像距增大和虚像距减小中的至少一种，Q为大于或者等于1，且小于或者等于X的整数。
请参阅图15，图15是本申请实施例提供的一种畸变类型与AR成像效果的示意图。各种类型的畸变处理以及得到的AR成像效果如图15所示，显然，在经过一定程度的径向畸变、切向畸变、虚像距增大、虚像距减小以及多次多项式(混合多种畸变类型)等畸变处理后，得到AR图像相较于原始的AR图像存在较为明显的畸变，严重影响用户的视觉感受和使用体验，大大降低驾驶的舒适性。其中，如图15所示，在虚像距变化的过程中，其显示的AR图像会产生一定程度的失焦模糊。需要说明的是，图15中的数学模型仅作为示例性说明，本申请实施例对此不作具体限定。请参阅图16，图16是本申请实施例提供的一种AR成像效果的对比示意图。如图16所示，经多次多项式畸变后的AR成像效果明显降低。可选地，可以根据畸变函数或畸变表的像素位置对应关系，叠加在将要输出至渲染管线的帧缓冲中，根据畸变的复杂度可以进行多次叠加，经过畸变处理后的AR成像效果可以直观地显示于虚拟化的HUD虚像面中。如此，由于这种直观性，本申请实施例进一步地相当于提供了一个平台供开发人员对畸变函数进行分析，以对HUD的去畸变功能进行研究开发，不断提升HUD的性能，保证用户的使用体验。
可选地，若在对该HUD的去畸变功能进行研究开发的过程中，需要对其中的去畸变算法进行测试时，则进一步还可以基于该去畸变算法，对该Q个第二AR图像进行去畸变处理，获取对应的Q个第三AR图像；融合该Q个第一车外场景和该Q个第三AR图像，以获得对应的Q个第四场景，并使能显示屏显示该Q个第四场景。然后，基于该Q个第三场景和该Q 个第四场景，对比去畸变处理前后的AR成像效果，对该去畸变算法的去畸变效果进行分析，进一步可以对该去畸变算法进行相应的修正，从而不断改进HUD的去畸变功能，等等，此处不再进行赘述。如此，在研究去畸变模型(或者说去畸变算法)的过程中，任何关于去畸变模型的参数变化，都可以以所见即所得的方式获得相应的效果，极大程度上提升了去畸变功能开发和测试的效率。
除此之外，驾驶过程中的抖动也会带来AR精度的退化。其中，抖动因素一般可以包括驾驶员人眼位置的抖动变化以及车辆行驶过程中造成的HUD抖动等，人眼位置的抖动和HUD的抖动均可以导致HUD投射的虚像(例如AR导航指引箭头等AR图标)和实物(也即真实世界中的物体，例如路面、行人和车辆等)存在一定的对齐失真。并且，不同驾驶员的身高不同，也容易导致个别驾驶员的观测位置与眼盒位置产生偏离，使其观测到的AR图标与真实世界的物体无法贴合，造成视觉上的失真，影响AR体验。相应的，为了缓解行驶过程中不可避免的抖动情况所带来的AR精度退化，HUD防抖动功能的开发就显得尤为重要。然而，在实车测试过程中，由于实际驾驶通常会伴随有多种抖动噪声(例如同时存在人眼位置抖动和HUD抖动)对AR的成像效果存在影响，因此很难抽离单个因素对其进行测试建模分析，也就难以高效、准确的对防抖动功能进行开发测试以及优化。
进一步地，本申请实施例还可以基于上述软件仿真的方法，模拟在实际行车过程中驾驶员的视线漂移和路面颠簸等对AR成像效果(或者说AR成像精度)带来的影响。具体可以在构建仿真场景中引入旋转位移量T[R|t]和抖动量(jitter，J)等，从而针对A成像效果的变化进行分析，为AR引擎防抖算法的开发提供有效的测试依据。请参阅图17，图17是本申请实施例提供的一种人眼视线坐标系和HUD虚像面坐标系的示意图。其中，人眼视线坐标系和HUD虚像面坐标系均可以基于固定的车身坐标系建立，可选地，也可以将人眼视线坐标系作为车身坐标系，或者，将HUD虚像面坐标系作为车身坐标系，等等，本申请实施例对此不作具体限定。请参阅图18，图18是本申请实施例提供的一种抖动情况对AR成像效果影响的示意图。如图18所示，当人眼位置存在位移量以及HUD虚像面位置存在抖动量的情况下，观测得到的AR图像与场景之间会存在一定程度的对齐失真，也即AR图标与相应的场景元素之间位置存在偏差，无法贴合，从而严重影响用户的视觉感受和使用体验。
可选地，还可以通过计算设备对S个第一车外场景对应的S个第一AR图像进行抖动处理，获取对应的S个第二AR图像；融合该S个第一车外场景和该S个第一AR图像，以获得对应的S个第三场景，并使能显示屏显示该S个第三场景。其中，该抖动处理可以包括叠加预设的旋转位移量和/或抖动量，S为大于或者等于1，且小于或者等于X的整数。
可选地，请一并参考上述图17和图18，一般情况下，人眼姿态通常可以抽象为一个六维的向量，在车身坐标系固定后，人眼相对于坐标系原点的向量可以表示为[t x，t y，t z|θ x，θ y，θ z]，其中，[t x，t y，t z]可以表示人眼相对于车身坐标系原点的位置偏移量，[θ x，θ y，θ z]可以表示人眼视线相对于坐标系轴的偏转角[yaw(偏航角)，pitch(俯仰角)，yoll(滚动角)]。
可选地，可以通过下述转移矩阵的方式来表示上述各量：
旋转矩阵：
Figure PCTCN2021084488-appb-000006
由此，得到转移矩阵：
Figure PCTCN2021084488-appb-000007
该转移矩阵可以对应于上述场景构建过程中的视角矩阵中的各个量，经过一系列的换算即可计算得到人眼视线在不同情况下的视角矩阵，进而得到虚拟HUD的正确观测视角。可以理解的是，针对HUD的抖动量，由于在行车过程中通常只存在上下的抖动，在分析过程中可以理想化该变量，将六自由度的变化抽象为上下的抖动量，并将该抖动量作为反馈量加入到上述转移矩阵中，最终体现至视角矩阵中，渲染引擎可以根据视角矩阵的变化，调整相应的观测效果，也即得到不同的AR成像效果，从而可以直观地针对AR成像效果的变化进行分析，支撑后续对HUD的去畸变功能进行研究开发，以不断提升HUD的性能，保证用户的使用体验。
可选地，若在对该HUD的防抖动功能进行研究开发的过程中，需要对其中的防抖动算法进行测试时，则进一步还可以基于该防抖动算法，对上述S个第二AR图像进行防抖动处理，获取对应的S个第三AR图像；融合该S个第一车外场景和该S个第三AR图像，以获得对应的S个第四场景，并使能显示屏显示该S个第四场景。然后，基于该S个第三场景和该S个第四场景，对比防抖动处理前后的AR成像效果，对该防抖动算法的防抖动效果进行分析，进一步可以对该防抖动算法进行相应的修正，从而不断改进HUD的防抖动功能。
可选地，可以理解的是，由于在实际使用过程中，畸变和抖动往往会同时存在，因此本申请实施例还可以对第一AR图像同时进行畸变处理以及抖动处理，以直观地得到在畸变和抖动同时影响下的AR成像效果，相应的，还可以对第二AR图像同时进行去畸变处理以及放抖动处理，以分析在畸变和抖动同时影响下的去畸变效果以及防抖动效果，等等，本申请实施例对此不作具体限定。
综上，通过本申请实施例提供的一种场景处理方法，首先，可以利用现有渲染引擎构建仿真场景或通过提前采集的真实道路数据构建实景仿真场景，并构建虚拟HUD的屏幕(也即HUD虚像面)于场景内，在HUD虚像面中绘制AR图标，从而可以直观地验证AR功能的正确性，为驾驶安全和舒适提供先验保障。由于场景可以任意合成、循环复用，使得测试覆盖率大大增加，一定程度上解决了现有技术中的AR功能体验和测试的场景难还原和不可重复性的问题。如此，本申请实施例可以支持多路况、多天气、多地区、多场景的可重复性，摆脱了硬件设备的制约，简化了AR功能的体验和测试流程，有助于模型、算法开发的快速迭代。其次，现有技术中，针对HUD的AR效果测试主要还是采用线下标定测量，通过驾驶实车对算法功能进行测试并改进，极大程度上依赖于离线数据的测试收集，因此无法还原AR功能失败(例如未显示正确的AR图标或者整个AR图像失真畸变)的场景。此外，在HUD标定安装后，参数已经基本确定，如遇安装偏差、挡风玻璃制造缺陷等问题，对于HUD成像效果的影响不可避免，从而容易导致测试精度低、效率低且投入成本高。而本申请实施例可以通过对畸变因素、多模态抖动因素的参数化模拟，使得这些外部因素对AR成像的影响可以直接反映在HUD虚像面中，可以直观地观察测定各类因素对HUD的AR成像效果的影响，有助于HUD的AR成像效果的稳定性分析。如此，本申请实施例提供的一种场景处理方法可以极大程度上提高测试效率，具有覆盖性广、重用性高、成本低的优点，并且能够支持增量场景、参数引入的扩展要求。
此外，需要说明的是，虚拟化仿真技术可以适用于很多方面，其核心思想是可以在虚拟的三维环境中构建关于真实世界的一切，因此，在一些可能的实施例中，除了虚拟AR-HUD硬件设备，仿真在驾驶过程中触发的AR功能事件在HUD设备上的显示效果外，还可以基于不同的需求仿真各类场景以及其他设备，从而实现高效、低成本地对其他设备进行测试分析，等等。
请参阅图19，图19是本申请实施例提供的一种场景处理装置的结构示意图，该场景处理装置30可以应用于上述计算设备。该场景处理装置30可以包括第一获取单元301、第二获取单元302、融合单元303和第一显示单元304，其中，各个单元的详细描述如下：
第一获取单元301，用于获取第一车外场景；其中，所述第一车外场景为二维的场景图像或者三维的场景模型；
第二获取单元302，用于获取与所述第一车外场景对应的第一AR图像；
融合单元303，用于融合所述第一车外场景和所述第一AR图像，以获得第二场景；其中，所述第一车外场景为所述第二场景中的现实信息，所述第一AR图像为所述第二场景中的虚拟信息；
第一显示单元304，用于使能显示屏显示所述第二场景。
在一种可能的实现方式中，所述第二获取单元302，具体用于：
根据所述第一车外场景与预设模型，获取与所述第一车外场景对应的所述第一AR图像；其中，所述第一AR图像包括一个或多个AR图标。
在一种可能的实现方式中，所述预设模型为神经网络模型，所述神经网络模型是根据由多个场景、多个AR图标以及所述多个场景和所述多个AR图标的不同匹配度训练得到的。
在一种可能的实现方式中，所述融合单元303，具体用于：
基于预设的抬头显示器HUD参数集合，在所述第一车外场景中确定对应的HUD虚像面；所述HUD虚像面为所述第一车外场景中对应的一个区域；
将所述第一AR图像渲染至所述HUD虚像面中，以获得第二场景。
在一种可能的实现方式中，所述HUD参数集合包括挡风玻璃曲率、眼盒位置、人眼观测位置、HUD安装位置和HUD虚像面尺寸中的至少一种参数。
在一种可能的实现方式中，所述第一获取单元301，具体用于：
获取第一传感器采集的数据；所述第一传感器为车载传感器；所述第一传感器采集的数据为在目标车辆行驶过程中针对所述目标车辆的周围环境采集的数据，包括图像数据、点云数据、温度数据和湿度数据中的至少一种；所述第一传感器包括摄像头、激光雷达、毫米波雷达、温度传感器和湿度传感器中的至少一种；
基于所述第一传感器采集的数据，构建所述第一车外场景，所述第一车外场景为实景仿真场景。
在一种可能的实现方式中，所述第一获取单元301，具体用于：
获取第二传感器采集的数据；所述第二传感器为通过预设的仿真模拟系统构建的传感器；所述第二传感器采集的数据为通过所述预设的仿真模拟系统设置的数据，包括天气、道路、行人、车辆、植物和交通信号中的至少一种数据；
基于所述第二传感采集的数据，构建所述第一车外场景，所述第一车外场景为虚拟仿真场景。
在一种可能的实现方式中，所述装置30还包括：
第一预处理单元305，用于对所述第一AR图像进行第一预处理，获取与所述第一车外场景对应的第二AR图像；所述第一预处理包括畸变处理和抖动处理中的至少一种；所述畸变处理包括径向畸变、切向畸变、虚像距增大和虚像距减小中的至少一种；所述抖动处理包括叠加预设的旋转位移量和/或抖动量；
第二融合单元306，用于融合所述第一车外场景和所述第二AR图像，以获得第三场景；
第二显示单元307，用于使能所述显示屏显示所述第三场景。
在一种可能的实现方式中，所述装置30还包括：
第二预处理单元308，用于对所述第二AR图像进行第二预处理，获取与所述第一车外场景对应的第三AR图像；所述第二预处理包括去畸变处理和防抖动处理中的至少一种；
第三融合单元309，用于融合所述第一车外场景和所述第三AR图像，以获得第四场景；
第三显示单元310，用于使能所述显示屏显示所述第四场景。
在一种可能的实现方式中，所述装置30还包括：
优化单元311，用于基于所述第三场景和所述第四场景，获取所述去畸变处理和/或所述防抖动处理的处理效果，以优化相应的去畸变功能和/或防抖动功能。
在一种可能的实现方式中，所述第一车外场景包括一个或多个场景元素；所述一个或多个场景元素包括天气、道路、行人、车辆、植物和交通信号中的一个或多个；所述一个或多个AR图标包括左转、右转和直行导航标识中的一个或多个；所述装置30还包括：
修正单元312，用于基于所述每个第二场景中的所述一个或多个场景元素以及所述一个或多个AR图标之间的位置关系和/或逻辑关系，对所述预设模型进行相应的修正。
其中，第一获取单元301用于执行上述图6a对应方法实施例中的步骤S801；第二获取单元302用于执行上述图6a对应方法实施例中的步骤S802；融合单元303用于执行上述图6a对应方法实施例中的步骤S8/3；第一显示单元304用于执行上述图6a对应方法实施例中的步骤S804；第一预处理单元305、第二融合单元306、第二显示单元307、第二预处理单元308、第三融合单元309、第三显示单元310、优化单元311和修正单元312用于执行上述图6a对应方法实施例中的步骤S804；
需要说明的是，本申请实施例中所描述的场景处理装置中各功能单元的功能具体可以参考上述图6a中所述的方法实施例中步骤S801-步骤S804的相关描述，还可以参考上述图6b中所述的方法实施例中步骤S901-步骤S903的相关描述，还可以参考上述图7中所述的方法实施例中步骤S11-步骤S15的相关描述，此处不再进行赘述。
图19中每个单元可以以软件、硬件、或其结合实现。以硬件实现的单元可以包括路及电炉、算法电路或模拟电路等。以软件实现的单元可以包括程序指令，被视为是一种软件产品，被存储于存储器中，并可以被处理器运行以实现相关功能，具体参见之前的介绍。
基于上述方法实施例以及装置实施例的描述，本申请实施例还提供一种计算设备。请参阅图20，图20是本申请实施例提供的一种计算设备的结构示意图。可选地，该计算设备1000可以为上述图3a、图3b和图3c中的计算设备100，其中，该计算设备1000至少包括处理器1001，输入设备1002、输出设备1003、计算机可读存储介质1004、数据库1005和存储器1006，该计算设备1000还可以包括其他通用部件，在此不再详述。其中，计算设备1000内 的处理器1001，输入设备1002、输出设备1003和计算机可读存储介质1004可通过总线或其他方式连接。
处理器1001可以用于实现场景处理装置30中的第一获取单元301、第二获取单元302、融合单元303、第一显示单元304、第一预处理单元305、第二融合单元306、第二显示单元307、第二预处理单元308、第三融合单元309、第三显示单元310、优化单元311和修正单元312，其中，实现过程的技术细节，具体可参考上述图6a中所述的方法实施例中步骤S801-步骤S803的相关描述，还可以参考上述图6b中所述的方法实施例中步骤S901-步骤S903的相关描述，还可以参考上述图7中所述的方法实施例中步骤S11-步骤S15的相关描述，此处不再进行赘述。其中，处理器1001可以是通用中央处理器(CPU)，微处理器，特定应用集成电路(application-specific integrated circuit，ASIC)，或一个或多个用于控制以上方案程序执行的集成电路。
该计算设备1000内的存储器1006可以是只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact Disc Read-Only Memory，CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。存储器1006可以是独立存在，通过总线与处理器1001相连接。存储器1006也可以和处理器1001集成在一起。
计算机可读存储介质1004可以存储在计算设备1000的存储器1006中，所述计算机可读存储介质1004用于存储计算机程序，所述计算机程序包括程序指令，所述处理器1001用于执行所述计算机可读存储介质1004存储的程序指令。处理器1001(或称CPU(Central Processing Unit，中央处理器))是计算设备1000的计算核心以及控制核心，其适于实现一条或一条以上指令，具体适于加载并执行一条或一条以上指令从而实现相应方法流程或相应功能；在一个实施例中，本申请实施例所述的处理器1001可以用于进行获取第一车外场景；其中，所述第一车外场景为二维的场景图像或者三维的场景模型；获取与所述第一车外场景对应的第一AR图像；融合所述第一车外场景和所述第一AR图像，以获得第二场景；其中，所述第一车外场景为所述第二场景中的现实信息，所述第一AR图像为所述第二场景中的虚拟信息；使能显示屏显示所述第二场景，等等。
本申请实施例还提供了一种计算机可读存储介质(Memory)，所述计算机可读存储介质是计算设备1000中的记忆设备，用于存放程序和数据。可以理解的是，此处的计算机可读存储介质既可以包括计算设备1000中的内置存储介质，当然也可以包括计算设备1000所支持的扩展存储介质。计算机可读存储介质提供存储空间，该存储空间存储了计算设备1000的操作系统。并且，在该存储空间中还存放了适于被处理器1001加载并执行的一条或一条以上的指令，这些指令可以是一个或一个以上的计算机程序(包括程序代码)。需要说明的是，此处的计算机可读存储介质可以是高速RAM存储器，也可以是非不稳定的存储器(non-volatile memory)，例如至少一个磁盘存储器；可选地还可以是至少一个位于远离前述处理器的计算机可读存储介质。
本申请实施例还提供一种计算机程序，该计算机程序包括指令，当该计算机程序被计算机执行时，使得计算机可以执行任意一种场景处理方法的部分或全部步骤。
需要说明的是，本申请实施例中所描述的计算设备1000中各功能单元的功能可参见上述图6a中所述的方法实施例中的步骤S801-步骤S804的相关描述，还可以参考上述图6b中所述的方法实施例中步骤S901-步骤S903的相关描述，还可以参见上述图7中所述的方法实施例中步骤S11-步骤S15的相关描述，此处不再赘述。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其它实施例的相关描述。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请并不受所描述的动作顺序的限制，因为依据本申请，某些步骤可能可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请所必须的。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置，可通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如上述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性或其它的形式。
上述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
上述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以为个人计算机、服务端或者网络设备等，具体可以是计算机设备中的处理器)执行本申请各个实施例上述方法的全部或部分步骤。其中，而前述的存储介质可包括：U盘、移动硬盘、磁碟、光盘、只读存储器(Read-OnlyMemory，缩写：ROM)或者随机存取存储器(RandomAccessMemory，缩写：RAM)等各种可以存储程序代码的介质。
此外，本申请的说明书和权利要求书及所述附图中的术语“第一”、“第二”、“第三”和“第四”等是用于区别不同对象，而不是用于描述特定顺序。此外，术语“包括”和“具有”以及它们任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有限定于已列出的步骤或单元，而是可选地还包括没有列出的步骤或单元，或可选地还包括对于这些过程、方法、产品或设备固有的其它步骤或单元。
在本文中提及“实施例”意味着，结合实施例描述的特定特征、结构或特性可以包含在本 申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例，也不是与其它实施例互斥的独立的或备选的实施例。本领域技术人员显式地和隐式地理解的是，本文所描述的实施例可以与其它实施例相结合。
需要说明的是，在本说明书中使用的术语“部件”、“模块”、“系统”等用于表示计算机相关的实体、硬件、固件、硬件和软件的组合、软件、或执行中的软件。例如，部件可以是但不限于，在处理器上运行的进程、处理器、对象、可执行文件、执行线程、程序和/或计算机。通过图示，在终端设备上运行的应用和终端设备都可以是部件。一个或多个部件可驻留在进程和/或执行线程中，部件可位于一个计算机上和/或分布在2个或更多个计算机之间。此外，这些部件可从在上面存储有各种数据结构的各种计算机可读介质执行。部件可例如根据具有一个或多个数据分组(例如来自与本地系统、分布式系统和/或网络间的另一部件交互的二个部件的数据，例如通过信号与其它系统交互的互联网)的信号通过本地和/或远程进程来通信。
以上所述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (26)

  1. 一种场景处理方法，其特征在于，包括：
    获取第一车外场景；其中，所述第一车外场景为二维的场景图像或者三维的场景模型；
    获取与所述第一车外场景对应的第一增强现实AR图像；
    融合所述第一车外场景和所述第一AR图像，以获得第二场景；其中，所述第一车外场景为所述第二场景中的现实信息，所述第一AR图像为所述第二场景中的虚拟信息；
    使能显示屏显示所述第二场景。
  2. 根据权利要求1所述的方法，其特征在于，所述获取与所述第一车外场景对应的第一AR图像，包括：
    根据所述第一车外场景与预设模型，获取与所述第一车外场景对应的所述第一AR图像；其中，所述第一AR图像包括一个或多个AR图标。
  3. 根据权利要求2所述的方法，其特征在于，所述预设模型为神经网络模型，所述神经网络模型是根据由多个场景、多个AR图标以及所述多个场景和所述多个AR图标的不同匹配度训练得到的。
  4. 根据权利要求2和3任意一项所述的方法，其特征在于，所述融合所述第一车外场景和所述第一AR图像，以获得第二场景，包括：
    基于预设的抬头显示器HUD参数集合，在所述第一车外场景中确定对应的HUD虚像面；所述HUD虚像面为所述第一车外场景中对应的一个区域；
    将所述第一AR图像渲染至所述HUD虚像面中，以获得第二场景。
  5. 根据权利要求4所述的方法，其特征在于，所述HUD参数集合包括挡风玻璃曲率、眼盒位置、人眼观测位置、HUD安装位置和HUD虚像面尺寸中的至少一种参数。
  6. 根据权利要求1-5任意一项所述的方法，其特征在于，所述获取第一车外场景，包括：
    获取第一传感器采集的数据；所述第一传感器为车载传感器；所述第一传感器采集的数据为在目标车辆行驶过程中针对所述目标车辆的周围环境采集的数据，包括图像数据、点云数据、温度数据和湿度数据中的至少一种；所述第一传感器包括摄像头、激光雷达、毫米波雷达、温度传感器和湿度传感器中的至少一种；
    基于所述第一传感器采集的数据，构建所述第一车外场景，所述第一车外场景为实景仿真场景。
  7. 根据权利要求1-5任意一项所述的方法，其特征在于，所述获取第一车外场景，包括：
    获取第二传感器采集的数据；所述第二传感器为通过预设的仿真模拟系统构建的传感器；所述第二传感器采集的数据为通过所述预设的仿真模拟系统设置的数据，包括天气、道路、行人、车辆、植物和交通信号中的至少一种数据；
    基于所述第二传感采集的数据，构建所述第一车外场景，所述第一车外场景为虚拟仿真 场景。
  8. 根据权利要求1-7任意一项所述的方法，其特征在于，所述方法还包括：
    对所述第一AR图像进行第一预处理，获取与所述第一车外场景对应的第二AR图像；所述第一预处理包括畸变处理和抖动处理中的至少一种；所述畸变处理包括径向畸变、切向畸变、虚像距增大和虚像距减小中的至少一种；所述抖动处理包括叠加预设的旋转位移量和/或抖动量；
    融合所述第一车外场景和所述第二AR图像，以获得第三场景；
    使能所述显示屏显示所述第三场景。
  9. 根据权利要求8所述的方法，其特征在于，所述方法还包括：
    对所述第二AR图像进行第二预处理，获取与所述第一车外场景对应的第三AR图像；所述第二预处理包括去畸变处理和防抖动处理中的至少一种；
    融合所述第一车外场景和所述第三AR图像，以获得第四场景；
    使能所述显示屏显示所述第四场景。
  10. 根据权利要求9所述的方法，其特征在于，所述方法还包括：
    基于所述第三场景和所述第四场景，获取所述去畸变处理和/或所述防抖动处理的处理效果，以优化相应的去畸变功能和/或防抖动功能。
  11. 根据权利要求2-10任意一项所述的方法，其特征在于，所述第一车外场景包括一个或多个场景元素；所述一个或多个场景元素包括天气、道路、行人、车辆、植物和交通信号中的一个或多个；所述一个或多个AR图标包括左转、右转和直行导航标识中的一个或多个；所述方法还包括：
    基于所述每个第二场景中的所述一个或多个场景元素以及所述一个或多个AR图标之间的位置关系和/或逻辑关系，对所述预设模型进行相应的修正。
  12. 一种场景处理装置，其特征在于，包括：
    第一获取单元，用于获取第一车外场景；其中，所述第一车外场景为二维的场景图像或者三维的场景模型；
    第二获取单元，用于获取与所述第一车外场景对应的第一AR图像；
    融合单元，用于融合所述第一车外场景和所述第一AR图像，以获得第二场景；其中，所述第一车外场景为所述第二场景中的现实信息，所述第一AR图像为所述第二场景中的虚拟信息；
    第一显示单元，用于使能显示屏显示所述第二场景。
  13. 根据权利要求12所述的装置，其特征在于，所述第二获取单元，具体用于：
    根据所述第一车外场景与预设模型，获取与所述第一车外场景对应的所述第一AR图像；其中，所述第一AR图像包括一个或多个AR图标。
  14. 根据权利要求13所述的装置，其特征在于，所述预设模型为神经网络模型，所述神经网络模型是根据由多个场景、多个AR图标以及所述多个场景和所述多个AR图标的不同匹配度训练得到的。
  15. 根据权利要求13和14任意一项所述的装置，其特征在于，所述融合单元，具体用于：
    基于预设的抬头显示器HUD参数集合，在所述第一车外场景中确定对应的HUD虚像面；所述HUD虚像面为所述第一车外场景中对应的一个区域；
    将所述第一AR图像渲染至所述HUD虚像面中，以获得第二场景。
  16. 根据权利要求15所述的装置，其特征在于，所述HUD参数集合包括挡风玻璃曲率、眼盒位置、人眼观测位置、HUD安装位置和HUD虚像面尺寸中的至少一种参数。
  17. 根据权利要求12-16任意一项所述的装置，其特征在于，所述第一获取单元，具体用于：
    获取第一传感器采集的数据；所述第一传感器为车载传感器；所述第一传感器采集的数据为在目标车辆行驶过程中针对所述目标车辆的周围环境采集的数据，包括图像数据、点云数据、温度数据和湿度数据中的至少一种；所述第一传感器包括摄像头、激光雷达、毫米波雷达、温度传感器和湿度传感器中的至少一种；
    基于所述第一传感器采集的数据，构建所述第一车外场景，所述第一车外场景为实景仿真场景。
  18. 根据权利要求12-16任意一项所述的装置，其特征在于，所述第一获取单元，具体用于：
    获取第二传感器采集的数据；所述第二传感器为通过预设的仿真模拟系统构建的传感器；所述第二传感器采集的数据为通过所述预设的仿真模拟系统设置的数据，包括天气、道路、行人、车辆、植物和交通信号中的至少一种数据；
    基于所述第二传感采集的数据，构建所述第一车外场景，所述第一车外场景为虚拟仿真场景。
  19. 根据权利要求12-18任意一项所述的装置，其特征在于，所述装置还包括：
    第一预处理单元，用于对所述第一AR图像进行第一预处理，获取与所述第一车外场景对应的第二AR图像；所述第一预处理包括畸变处理和抖动处理中的至少一种；所述畸变处理包括径向畸变、切向畸变、虚像距增大和虚像距减小中的至少一种；所述抖动处理包括叠加预设的旋转位移量和/或抖动量；
    第二融合单元，用于融合所述第一车外场景和所述第二AR图像，以获得第三场景；
    第二显示单元，用于使能所述显示屏显示所述第三场景。
  20. 根据权利要求19所述的装置，其特征在于，所述装置还包括：
    第二预处理单元，用于对所述第二AR图像进行第二预处理，获取与所述第一车外场景对应的第三AR图像；所述第二预处理包括去畸变处理和防抖动处理中的至少一种；
    第三融合单元，用于融合所述第一车外场景和所述第三AR图像，以获得第四场景；
    第三显示单元，用于使能所述显示屏显示所述第四场景。
  21. 根据权利要求20所述的装置，其特征在于，所述装置还包括：
    优化单元，用于基于所述第三场景和所述第四场景，获取所述去畸变处理和/或所述防抖动处理的处理效果，以优化相应的去畸变功能和/或防抖动功能。
  22. 根据权利要求13-21任意一项所述的装置，其特征在于，所述第一车外场景包括一个或多个场景元素；所述一个或多个场景元素包括天气、道路、行人、车辆、植物和交通信号中的一个或多个；所述一个或多个AR图标包括左转、右转和直行导航标识中的一个或多个；所述装置还包括：
    修正单元，用于基于所述每个第二场景中的所述一个或多个场景元素以及所述一个或多个AR图标之间的位置关系和/或逻辑关系，对所述预设模型进行相应的修正。
  23. 一种场景处理系统，其特征在于，包括：终端和服务器；
    所述终端用于发送第一车外场景；其中，所述第一车外场景为所述终端的传感器获取的传感信息；
    所述服务器用于接收来自于所述终端的所述第一车外场景；
    所述服务器还用于获取与所述第一车外场景对应的第一AR图像；
    所述服务器还用于融合所述第一车外场景和所述第一AR图像，以获得第二场景；其中，所述第一车外场景为所述第二场景中的现实信息，所述第一AR图像为所述第二场景中的虚拟信息；
    所述服务器还用于发送所述第二场景；
    所述终端还用于接收所述第二场景，并显示所述第二场景。
  24. 一种计算设备，其特征在于，包括处理器和存储器，所述处理器和存储器相连，其中，所述存储器用于存储程序代码，所述处理器用于调用所述程序代码，以执行如权利要求1至11任意一项所述的方法。
  25. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机程序，该计算机程序被处理器执行时实现上述权利要求1至11任意一项所述的方法。
  26. 一种计算机程序，其特征在于，所述计算机程序包括指令，当所述计算机程序被计算机执行时，使得所述计算机执行如权利要求1至11任意一项所述的方法。
PCT/CN2021/084488 2021-03-31 2021-03-31 一种场景处理方法、装置、系统及相关设备 WO2022205102A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2021/084488 WO2022205102A1 (zh) 2021-03-31 2021-03-31 一种场景处理方法、装置、系统及相关设备
CN202180001054.8A CN113260430B (zh) 2021-03-31 2021-03-31 一种场景处理方法、装置、系统及相关设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/084488 WO2022205102A1 (zh) 2021-03-31 2021-03-31 一种场景处理方法、装置、系统及相关设备

Publications (1)

Publication Number Publication Date
WO2022205102A1 true WO2022205102A1 (zh) 2022-10-06

Family

ID=77180569

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/084488 WO2022205102A1 (zh) 2021-03-31 2021-03-31 一种场景处理方法、装置、系统及相关设备

Country Status (2)

Country Link
CN (1) CN113260430B (zh)
WO (1) WO2022205102A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116311935A (zh) * 2023-03-20 2023-06-23 冉林甫 一种基于大数据的智慧城市交通管理方法

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114141092B (zh) * 2021-11-10 2023-01-20 武汉未来幻影科技有限公司 一种驾考模拟器动画场景构建方法及系统
CN114299162B (zh) * 2021-12-30 2024-05-10 合众新能源汽车股份有限公司 一种用于ar-hud的快速标定方法
CN114661398B (zh) * 2022-03-22 2024-05-17 上海商汤智能科技有限公司 一种信息展示方法、装置、计算机设备和存储介质
CN114859754B (zh) * 2022-04-07 2023-10-03 江苏泽景汽车电子股份有限公司 一种抬头显示系统的仿真测试方法和仿真测试系统
CN116774679B (zh) * 2023-08-25 2023-11-28 北京斯年智驾科技有限公司 一种自动驾驶车辆测试方法、系统、装置和存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130293582A1 (en) * 2012-05-07 2013-11-07 Victor Ng-Thow-Hing Method to generate virtual display surfaces from video imagery of road based scenery
CN104266654A (zh) * 2014-09-26 2015-01-07 广东好帮手电子科技股份有限公司 一种车载实景导航系统及方法
CN107589851A (zh) * 2017-10-17 2018-01-16 极鱼（北京）科技有限公司 汽车的交互方法和系统
CN110525342A (zh) * 2019-08-30 2019-12-03 的卢技术有限公司 一种基于深度学习的ar-hud车载辅助驾驶方法及其系统
CN110663032A (zh) * 2017-12-21 2020-01-07 谷歌有限责任公司 加强对增强现实(ar)应用的测试的支持

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106383587B (zh) * 2016-10-26 2020-08-04 腾讯科技（深圳）有限公司 一种增强现实场景生成方法、装置及设备
CN207180987U (zh) * 2017-08-07 2018-04-03 安徽江淮汽车集团股份有限公司 用于平视显示装置的台架测试系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130293582A1 (en) * 2012-05-07 2013-11-07 Victor Ng-Thow-Hing Method to generate virtual display surfaces from video imagery of road based scenery
CN104266654A (zh) * 2014-09-26 2015-01-07 广东好帮手电子科技股份有限公司 一种车载实景导航系统及方法
CN107589851A (zh) * 2017-10-17 2018-01-16 极鱼（北京）科技有限公司 汽车的交互方法和系统
CN110663032A (zh) * 2017-12-21 2020-01-07 谷歌有限责任公司 加强对增强现实(ar)应用的测试的支持
CN110525342A (zh) * 2019-08-30 2019-12-03 的卢技术有限公司 一种基于深度学习的ar-hud车载辅助驾驶方法及其系统

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116311935A (zh) * 2023-03-20 2023-06-23 冉林甫 一种基于大数据的智慧城市交通管理方法
CN116311935B (zh) * 2023-03-20 2024-03-15 湖北省规划设计研究总院有限责任公司 一种基于大数据的智慧城市交通管理方法

Also Published As

Publication number Publication date
CN113260430B (zh) 2022-07-22
CN113260430A (zh) 2021-08-13

Similar Documents

Publication Publication Date Title
WO2022205102A1 (zh) 一种场景处理方法、装置、系统及相关设备
WO2021013193A1 (zh) 一种交通灯的识别方法及装置
JP6988819B2 (ja) 画像処理装置、および画像処理方法、並びにプログラム
US10600251B2 (en) Method of ground adjustment for in-vehicle augmented reality systems
US10410427B2 (en) Three dimensional graphical overlays for a three dimensional heads-up display unit of a vehicle
US10712556B2 (en) Image information processing method and augmented reality AR device
US20160210775A1 (en) Virtual sensor testbed
CN109961522B (zh) 图像投射方法、装置、设备和存储介质
WO2022204855A1 (zh) 一种图像处理方法及相关终端装置
US20220204009A1 (en) Simulations of sensor behavior in an autonomous vehicle
KR20190067366A (ko) 차량 및 그 제어 방법
CN111833443A (zh) 自主机器应用中的地标位置重建
CN114240769A (zh) 一种图像处理方法以及装置
US20230342883A1 (en) Image processing method and apparatus, and storage medium
CN111815745A (zh) 行驶状况展示方法、装置、存储介质和电子设备
CN114820504B (zh) 图像融合偏差的检测方法、装置、电子设备及存储介质
US20230196619A1 (en) Validation of virtual camera models
CN115205461B (zh) 场景重建方法、装置、可读存储介质及车辆
WO2024092559A1 (zh) 一种导航方法及相应装置
KR102482829B1 (ko) 차량용 ar 디스플레이 장치 및 ar 서비스 플랫폼
US20230005214A1 (en) Use of Real-World Lighting Parameters to Generate Virtual Environments
Gruyer et al. Proposal of a virtual and immersive 3D architecture dedicated for prototyping, test and evaluation of eco-driving applications
CN116933655A (zh) 基于车载内外混合现实的自动驾驶仿真方法及系统
Gałązka et al. CiThruS2: Open-Source Virtual Environment for Simulating Real-Time Drone Operations and Piloting
CN116228587A (zh) 一种智能3d倒车影像增强系统及方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21933772

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21933772

Country of ref document: EP

Kind code of ref document: A1