WO2023241139A1

WO2023241139A1 - Intelligent carriage control method, controller, intelligent carriage, and storage medium

Info

Publication number: WO2023241139A1
Application number: PCT/CN2023/081396
Authority: WO
Inventors: 张永亮
Original assignee: 中兴通讯股份有限公司
Priority date: 2022-06-13
Filing date: 2023-03-14
Publication date: 2023-12-21
Also published as: CN117261585A

Abstract

The present application discloses an intelligent carriage control method, a controller, an intelligent carriage, and a computer storage medium. The intelligent carriage control method comprises: acquiring fusion information data and communication data, the fusion information data comprising monitoring information obtained by a millimeter wave module scanning the inside of the intelligent carriage, and the communication data comprising communication data for interaction between the millimeter wave module and the inside and outside of the intelligent carriage (S610); obtaining augmented reality data according to the fusion information data and the communication data (S620); rendering the augmented reality data to obtain projection data (S630); and sending the projection data to a projection device for projection display (S640).

Description

Intelligent cockpit control method, controller, intelligent cockpit and storage medium

Cross-references to related applications

This application is filed based on a Chinese patent application with application number 202210659957.1 and a filing date of June 13, 2022, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated into this application as a reference.

Technical field

This application relates to the field of vehicle equipment control technology, and specifically relates to an intelligent cockpit control method, a controller, an intelligent cockpit and a computer storage medium.

Background technique

In recent years, the automobile industry has entered an era of intelligence and electrification. The development of smart cockpits has become a general trend. Compared with traditional car cockpits, smart cockpits can display information about the situation ahead on the windshield of the car for users through a vehicle-mounted head-up display projector. Traffic information to assist users in driving.

However, in smart cockpit technology, the current vehicle head-up display device has weak processing capabilities, resulting in the vehicle head-up display projector being unable to process more complex data when providing augmented reality services to users, and can only provide simple traffic information. Projection has a single function and cannot further provide users with the driving information or entertainment content they need. Moreover, it is limited by the processing power of the vehicle head-up display device. When the head-up display projector provides users with augmented reality services, it runs slowly and affects user experience.

Contents of the invention

Embodiments of the present application provide an intelligent cockpit control method, a controller, an intelligent cockpit and a computer storage medium.

In a first aspect, embodiments of the present application provide a smart cockpit control method, which is applied to the smart cockpit. The smart cockpit includes a millimeter wave module, and the millimeter wave module is connected to a projection device. The method includes: obtaining fusion Information data and communication data, the fused information data includes monitoring information obtained by scanning the interior of the smart cockpit by the millimeter wave module, and the communication data includes communication data of the interaction between the millimeter wave module and the inside and outside of the smart cockpit; according to the The information data and the communication data are fused to obtain augmented reality data; projection data is obtained after rendering processing of the augmented reality data; and the projection data is sent to the projection device for projection display.

In a second aspect, embodiments of the present application provide an intelligent cockpit controller, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program. The program implements the intelligent cockpit control method described in any embodiment of the first aspect.

In the third aspect, embodiments of the present application provide a smart cockpit, including the control system described in the second aspect. device.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium that stores computer-executable instructions. The computer-executable instructions are used to execute the smart cockpit control method described in any embodiment of the first aspect.

Description of the drawings

Figure 1 is a schematic diagram of a smart cockpit proposed by an embodiment of the present application;

Figure 2 is a functional block diagram of a smart cockpit proposed by another embodiment of the present application;

Figure 3 is an extended functional block diagram of a smart cockpit proposed by another embodiment of the present application;

Figure 4 is another extended functional block diagram of the smart cockpit proposed by another embodiment of the present application;

Figure 5 is a system architecture diagram of a smart cockpit proposed by another embodiment of the present application;

Figure 6 is a method flow chart of a smart cockpit control method proposed by another embodiment of the present application;

Figure 7 is a flow chart of a method in which the first millimeter wave unit obtains fusion information data in the smart cockpit control method proposed by another embodiment of the present application;

Figure 8 is a flowchart of a method corresponding to the second millimeter wave unit acquiring fusion information data in the smart cockpit control method proposed by another embodiment of the present application;

Figure 9 is a flow chart of a method for performing federated calculations by the first millimeter wave unit and the second millimeter wave unit in the smart cockpit control method proposed by another embodiment of the present application;

Figure 10 is a structural diagram of an intelligent cockpit controller proposed by another embodiment of this application.

Reference signs: 101. Antenna array; 102. Second millimeter wave unit; 103. Augmented reality glasses; 104. Handle; 105. Steering wheel; 106. First millimeter wave unit; 107. Driver monitoring system camera; 110. Head up Display projector; 111. Non-adjustable curved lens; 112. Digital light processing optical machine; 113. Adjustable curved lens.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the embodiments described here are only used to explain the present application and are not used to limit the present application.

In some embodiments, although the functional modules are divided in the system schematic diagram and the logical sequence is shown in the flow chart, in some cases, the module division in the system or the order in the flow chart may be different. Follow the steps shown or described. The terms first, second, etc. in the description, claims, and above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

This application discloses an intelligent cockpit control method, a controller, an intelligent cockpit and a computer storage medium. Through the intelligent cockpit control method proposed in this application, fusion information data is obtained. The fusion information data includes scanning of the interior of the intelligent cockpit by a millimeter wave module. For the obtained monitoring information, augmented reality data is obtained based on the fusion information data, and the projection data is obtained after rendering the augmented reality data. The millimeter wave module is used as the augmented reality generator, the augmented reality data is obtained based on the multi-sensor fusion data, and the augmented reality data is obtained. The data is rendered and processed to more quickly and stably provide users with augmented reality services through the vehicle-mounted augmented reality head-up display projector, display driving information and entertainment content to users according to user needs, and provide users with a more intelligent interactive experience.

The embodiments of the present application will be further described below with reference to the accompanying drawings.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of an intelligent cockpit proposed by an embodiment of the present application. The embodiment of the present application provides an intelligent cockpit.

In one implementation, as shown in FIG. 1 , a smart cockpit proposed by an embodiment of the present application includes: a head-up display screen projector 110 , a first millimeter wave unit 106 , a second millimeter wave unit 102 and a driver monitoring system camera 107 ,

In one implementation, the head-up display projector 110 is an AR-3D-HUD (Augmented Reality 3D Head Up Display) projector, the driver monitoring system camera 107 is a DMS (Driver monitoring system) camera, and the augmented reality glasses 103 They are AR (Augmented Reality) glasses.

In some embodiments, the smart cockpit proposed by an embodiment of the present application also includes augmented reality glasses 103, a handle 104 and a steering wheel 105, wherein the head-up display projector 110 includes a DLP digital light processing optical engine 112 (Digital Light Processing, digital Light processing optical machine), non-adjustable curved lenses 111 and adjustable curved lenses 113; among them, the first millimeter wave unit 106 is placed in the cockpit facing the driver and close to the head-up display projector 110, and is called a front millimeter wave module , the first millimeter wave unit 106 has a beamforming controllable millimeter wave antenna array 101.

In some embodiments, the first millimeter wave unit 106 and the second millimeter wave unit 102 monitor the driver's breathing and heartbeat by detecting the phase changes of frequency modulated continuous wave signals in a specific range of values caused by the target's tiny vibrations, and through the target object. Parameters such as distance, speed, horizontal angle and pitch angle can be used to perform 4D imaging processing on the driver's dangerous actions, distractions and fatigue driving conditions to assist the driver in safe driving.

In some embodiments, because the first millimeter wave unit 106 and the second millimeter wave unit 102 have stronger computing power, the first millimeter wave unit 106 and the second millimeter wave unit 102 replace the conventional heads-up display projector 110. The head-up display projector drive board serves as the AR algorithm processing center. The second millimeter wave unit 102 is placed inside the top of the cabin and is connected to the antenna array 101 outside the roof, which is called a top-mounted millimeter wave module. The overhead millimeter-wave module has a millimeter-wave antenna array 101 that can be controlled by beamforming both inside the cabin and outside the roof. On the one hand, it is used to monitor the breathing and heartbeat of passengers and conduct 4D imaging monitoring of passengers' behaviors. It is used in the cabin edge computing center and Content Delivery Network (Content Delivery Network) to provide powerful AR rendering computing power for AR data in the smart cockpit. At the same time, the second millimeter wave unit has streaming media content distribution capabilities. It can communicate with the road side unit (Road side unit), Internet of Vehicles and mobile Internet through the antenna array 101 on the outside of the car roof for V2X (vehicle to everything, vehicle to the outside world) communication. The front-mounted millimeter wave module and the top-mounted millimeter wave module communicate through millimeter waves in the typical frequency band above 60GHz. By virtue of the large bandwidth and low latency communication characteristics of millimeter waves, as well as point cloud imaging capabilities and beam phase difference positioning capabilities, the two During the communication process, 4D imaging perception and breathing and heart rate recognition of the driver and passengers can be simultaneously realized.

In some instances, DMS cameras are used for infrared observation of drivers and eye tracking of people in smart cockpits. At the same time, DMS cameras are infrared cameras that can work better at night.

Referring to Figures 1 and 2, Figure 2 is a functional block diagram of a smart cockpit proposed by another embodiment of the present application. This embodiment of the present application provides a smart cockpit.

In one embodiment, as shown in Figure 2, in the functional block diagram of the smart cockpit proposed in another embodiment, the infotainment host is a virtual functional module, and the model display function of the infotainment host is provided by the millimeter wave module 1 (i.e. the first One millimeter wave unit 106) or millimeter wave module 2 (i.e., the second millimeter wave unit 102) is responsible for the millimeter wave module 1 or millimeter wave module 2 to obtain the map navigation information from the positioning module, the DMS camera information from the DMS camera, and the vehicle body Vehicle body speed, acceleration and steering angle information from the sensor, ADAS information from the ADAS (Advanced Driving Assistance System) module, and 4D imaging and respiration, heart rate and other IMS (In -carbin monitoring system: includes DMS and OMS (occupancy monitoring system, passenger monitoring system) information. At the same time, millimeter wave module 1 or millimeter wave module 2 is used as an AR generator for data fusion algorithm processing to extract effective AR Information is fed into the display model.

In some embodiments, the AR generator of the infotainment host is the first millimeter wave unit 106, and the rendered AR display model information is sent to the head-up display projector via LVDS (Low-Voltage Differential Signaling) 110, in the head-up display projector 110 through the head-up display projector drive board, image generation unit, optical/structural components (the first two correspond to the digital light processing optical engine 112 in Figure 1, the latter corresponds to the non-adjustable The curved lens 111 and the adjustable curved lens 113) project into the human eye. However, what is seen from the perspective of the human eye is a virtual image formed by the windshield mirror at a certain distance in front of the car. Among them, the millimeter wave module 1 itself has to process the 4D imaging point cloud, so it has powerful computing power and can be used for AR generation in the infotainment host. The algorithm processing center of the processor.

In some embodiments, the second millimeter wave unit 102 not only has powerful computing power and powerful transmission capabilities, not only can be used as an AR renderer, but also has powerful data caching capabilities to store data copies to meet user needs nearby, and at the same time borrow Content updates are realized through V2X communication with external network spaces.

In some embodiments, the millimeter wave module can sense while communicating. The first millimeter wave unit 106 and the second millimeter wave unit 102 can perform communication between the two millimeter wave modules and with the head-up display projector. It can perform 4D imaging and heart rate and respiration detection while communicating before 110, and can provide full coverage for IMS including DMS and OMS.

In some embodiments, the second millimeter wave unit 102 implements external Internet of Vehicles via V2X communication for XR (extended-range, extended range) display in the car cockpit, audio-visual entertainment, real-time navigation equipment, and content of mobile terminals such as user mobile phones. Update, at the same time, the original system processor performs 4D imaging of people or objects in the car cockpit and the breathing and heartbeat monitoring data of the drivers and passengers in the car cockpit are shared to the roadside RSU (Road Side Unit, Road Side Unit) when safety permits. Side unit), Internet of Vehicles computing center and mobile Internet, associated individual or group end users can remotely understand the situation in the car in real time. At the same time, the large bandwidth and low latency of the millimeter wave module itself can improve AR rendering capabilities, thereby improving the image quality of the head-up display projector.

Referring to Figures 1 and 3, Figure 3 is an extended functional block diagram of an intelligent cockpit control proposed by another embodiment of the present application. An embodiment of the present application provides an intelligent cockpit.

Usually, car head-up display projectors are only used for AR display based on real-time navigation map information and vehicle condition information while the car is driving. However, as the degree of autonomous driving of cars is increasing day by day, and when the car is driving completely autonomously, head-up display Screen projection can also be used for user entertainment, so this application provides a smart cockpit that uses a millimeter wave module as an augmented reality generator to obtain augmented reality data based on multi-sensor fusion data, and renders the augmented reality data to make it more Quickly and stably provide users with augmented reality services through the vehicle head-up display projector, which can provide users with entertainment and metaverse experiences during autonomous driving.

In one implementation, as shown in Figure 3, the car head-up display projector 110 is used for metaverse experience, which needs to enhance the user's body perception and spatial positioning. At the user level, AR glasses and a handle 104 are added at the driver's position for The user increases spatial positioning through AR glasses and performs azimuth control through the handle 104; at the same time, the millimeter wave module uses 4D imaging to perceive the user's head freedom, gestures, expressions, and body movements, and enhances the DMS camera's refined tracking of the eyeballs , ensuring accurate calculations in terms of user body perception and spatial positioning, enabling virtual objects to be placed in the real world and perfectly integrated with the real world. In this process, based on the large bandwidth and low latency of millimeter waves and its own powerful computing power support, a superior experience environment of the metaverse in the car cockpit is built, making the virtual image environment relying on the vehicle head-up display projector even grander, making An even better immersive experience based on extended reality technology.

Referring to FIG. 1 and FIG. 4 , FIG. 4 is another extended functional block diagram of an intelligent cockpit proposed by another embodiment of the present application. This embodiment of the present application provides an intelligent cockpit.

In one embodiment, as shown in Figure 4, the smart cockpit in this application performs data interaction based on millimeter wave modules and associates with external networks, which can further improve the computing power of processing sensor information, and can make the information source object to be processed by the millimeter wave module It is further expanded to the aspects of taste, smell, touch, and consciousness to realize an interactive virtual reality combined with augmented reality and metaverse experience environment.

In some embodiments, the smart cockpit of the present application adds windshield-based HUD hardware at the passenger position of the smart cockpit, which can provide passengers with augmented reality services more quickly and stably through the vehicle head-up display projector, and provide augmented reality services according to user needs. Passengers are shown driving information and entertainment content.

In some embodiments, the smart cockpit of the present application includes an AR helmet, and the AR helmet provides users with an immersive experience through the on-board millimeter wave communication and environmental sensing functions of the smart cockpit.

In some embodiments, the fused information data also includes sensing information, and the augmented reality data obtained according to the fused information data includes obtaining the augmented reality data according to the monitoring information and the sensing information, wherein the sensing information includes at least one of the following: The user eye tracking data obtained by the driver monitoring camera, the user keystroke data obtained through the handle or the spatial positioning data obtained by the augmented reality glasses can realize an interactive virtual and real combined augmented reality and metaverse experience environment.

In some embodiments, the vehicle-mounted AR-HUD display generates a mirror of the real world based on digital twin technology. By adding blockchain technology to the edge computing center of the millimeter wave module, the AR-HUD fusion processing of millimeter wave communication and perception information in the car cockpit is achieved. HUD can well construct the economic, social, and social aspects between the virtual world and the real world. The environment that is closely integrated in terms of identity makes the car an important place for users to experience the metaverse.

The first millimeter wave unit and the second millimeter wave unit of the above-mentioned millimeter wave module may be two independent physical entities connected through millimeter wave communication, or they may be two virtual functional units within a common physical entity. The two physical entity embodiments are implemented by being placed on the driver's seat and the roof of the car respectively, and the common physical entity embodiment is implemented by being placed inside the car's interior rearview mirror. The millimeter wave common physical entity module is located at the rearview mirror in the car. This millimeter wave unit is used to provide an AR-HUD experience for the driver's position by monitoring the gestures, movements, and expressions of the user at this location, using only a millimeter. The wave physical entity module can save smart cockpit costs and smart cockpit space. However, at this time, the millimeter wave module continuously positions and dynamically controls the position of the monitored object very frequently. In the process of millimeter wave communication and perception information fusion processing, AR-HUD gradually During the upgrade iteration process, the computing power requirements for a single millimeter wave module are very high.

In some embodiments, the smart cockpit of the present application includes two physical millimeter wave units. The division of labor between the two millimeter wave units is more conducive to ensuring overall performance. Using two millimeter wave units, the first millimeter wave unit 106 is also used as a The algorithm processing center of the AR generator, the second millimeter wave unit 102 has the function of an edge computing center and also serves as an edge CDN (content delivery network, content distribution network), allowing users to quickly obtain the required content or game data nearby, thereby making it more convenient Quickly and stably provide users with augmented reality services through the vehicle head-up display projector, displaying driving information and entertainment content to users according to their needs.

Referring to FIG. 5 , FIG. 5 is a system architecture diagram of a smart cockpit proposed by another embodiment of the present application. This embodiment of the present application provides a smart cockpit.

In one implementation, as shown in Figure 5, considering the balance of computing power of each millimeter wave module and the fact that more millimeter wave modules may actually exist, and the respective functions of each millimeter wave module can be coordinated with each other, replaced or calculated collaboratively, for specific The function does not refer specifically to a specific millimeter wave module, but is uniformly expressed as millimeter wave module Xn.

In some embodiments, the first millimeter wave unit 106 serves as an AR generator for fusing computing data, AR rendering data, V2X communication data, CDN communication forwarding data, DMS sensing data, OMS sensing data and edge computing center data to obtain multi-sensor The fused data allows the second millimeter wave unit 102 to perform subsequent work based on the fused data, where each data in the above fused data interacts through the control entrance of the control bus.

In some embodiments, the control bus in Figure 5 also includes central control screen user UI control and user AR glasses and handle 104 control in addition to system communication control, system perception control and system calculation control. Compared with the first three, which belong to the millimeter The implicit control of the internal system of the module is invisible to the user within the wave module or between the millimeter wave modules. The latter two are visible controls at the user level, making the user's infotainment content choices in the car cockpit more abundant and rising to the meta level. The level of cosmic experience.

In some embodiments, the millimeter wave module of this application is actually an extension of the millimeter wave radar. In addition to the ordinary millimeter wave radar, a certain millimeter wave module also serves as an AR generator, and another millimeter wave module also serves as an edge computing center and CDN can improve users' in-vehicle AR-HUD experience by enriching the resources, capabilities and applications of smart cockpit communication and perception integration.

Referring to Figure 6, Figure 6 is a method flow chart of a smart cockpit control method proposed by another embodiment of the present application. The embodiment of the present application provides a smart cockpit control method, which is applied to the smart cockpit. The smart cockpit includes a millimeter wave module, and the millimeter wave module is connected to a head-up display screen projector. The method includes but is not limited to the following steps:

Step S610, obtain fusion information data and communication data. The fusion information data includes monitoring information obtained by scanning the interior of the smart cockpit by the millimeter wave module, and the communication data includes communication data about the interaction between the millimeter wave module and the inside and outside of the smart cockpit;

Step S620, obtain augmented reality data based on the fused information data and communication data;

Step S630, perform rendering processing on the augmented reality data to obtain projection data;

Step S640: Send the projection data to the projection device for projection display.

In one embodiment, the communication data includes at least one of the following: Internet of Vehicles data, content distribution network data or associated edge computing data acquired by the millimeter wave module inside and outside the smart cockpit, where the Internet of Vehicles data is the vehicle-mounted device through wireless communication Technology, vehicle dynamic information data such as driving road conditions and driving speed are obtained from external information network platforms. The millimeter wave module implements corresponding data display and intelligent driving functions based on the Internet of Vehicles data; the content distribution network data is passed through the millimeter wave module through the exterior of the car roof. The antenna array communicates between the vehicle and the outside world, and then obtains the communication data from the roadside unit. The millimeter wave module can update local rendering resources in real time based on the content distribution network data to better render the augmented reality data;

Edge computing data is communication data obtained by the millimeter wave module from the edge computing center inside the car. During the rendering process of the millimeter wave module, the edge computing center can share part of the computing work and send the calculation results in the form of edge computing data. To the millimeter wave module, thereby improving the rendering efficiency of the millimeter wave module.

In one embodiment, the projection device, that is, the head-up display screen projector, obtains fusion information data and communication data through the millimeter wave module. The fusion information data includes monitoring information obtained by scanning the interior of the smart cockpit by the millimeter wave module. The communication data includes the millimeter wave module. For the Internet of Vehicles data, content distribution network data and associated edge computing data acquired inside and outside the smart cockpit, augmented reality data is obtained based on the fusion information data and communication data, rendering demand information is generated based on the augmented reality data, and the augmented reality data is generated based on the rendering demand information. The data is rendered and processed to obtain the projection data, and the projection data is sent to the head-up display projector for projection display, and then the millimeter wave module is used as the augmented reality generator to obtain augmented reality data based on multi-sensor fusion data and communication data, and the augmented reality The data is rendered and processed to more quickly and stably provide users with augmented reality and metaverse services through the vehicle head-up display projector, which can display driving information and entertainment content to users according to their needs, allowing users to quickly obtain the information or games they need. data and display it.

In some embodiments, before obtaining the fusion information data, the method further includes scanning the cabin where the millimeter wave module is located to obtain a positioning calculation value, obtaining target location information based on the positioning calculation value, and obtaining monitoring information based on the target location information.

In some embodiments, the millimeter wave module is provided with a millimeter wave antenna array. Before obtaining the fusion information data, the method further includes: obtaining the phase interference difference value of the millimeter wave antenna array, obtaining the target position information according to the phase interference difference value, and obtaining the target position information according to the phase interference difference value. Get monitoring information.

In one embodiment, in order to obtain the action or physical sign information representing the driver or passenger, the position of the driver or passenger needs to be determined first. Therefore, a dedicated phase interference array can be selected in the millimeter wave antenna array. The phase interference difference of each element is used to determine the corresponding target position information of the driver or passenger; at the same time, the target position information corresponding to the driver or passenger can be determined by directly scanning the target object with millimeter waves and first performing positioning calculations. Positioning through millimeter waves can make the positioning The accuracy reaches centimeter level. Based on this level of precision positioning, various applications that require user location information in the smart cockpit can be realized, and then more accurately provide users with augmented reality services through the vehicle head-up display projector, and display driving information to users based on their location. and entertainment content.

In some embodiments, after positioning the target and obtaining the target position information, the millimeter wave module performs phase control on the millimeter wave antenna array to perform beam alignment based on the target position information, and repeatedly performs beam scanning and pairing actions during the alignment process. , ensuring that the sensing data of the object monitored by the millimeter wave module is updated in real time when the object monitored by the millimeter wave module changes dynamically.

In some embodiments, the millimeter wave module performs 4D imaging and breathing and heart rate monitoring respectively for the driver and the passenger. The 4D imaging performs distance, speed, horizontal angle and pitch angle imaging by imaging the millimeter wave point cloud reflected by the monitoring target. Analyze and identify human movements, breathing, and heart rate monitoring to detect the driver's and passengers' breathing using millimeter waves to detect the phase changes of the frequency modulated continuous wave signal CSI (Channel State Information) in a specific range caused by the tiny vibrations of the target. , heart rate monitoring.

In some embodiments, the millimeter wave module can perform 4D imaging and breathing and heart rate monitoring for the driver and passenger respectively in two ways: time-sharing sensing, concentrating on monitoring the target for a large period of time and then switching to breathing and breathing at regular intervals after 4D imaging of the target. Heart rate monitoring, among which, through regular monitoring during the time-sharing sensing process, the purpose of 4D imaging and respiration and heart rate monitoring for drivers and passengers is achieved respectively and computer resources are saved; Simultaneous sensing, due to the impact on the performance of millimeter wave modules through simultaneous sensing processing As requirements increase, millimeter wave modules are required to use advanced MIMO waveforms to design multiple antenna channels to work simultaneously, use software algorithms to achieve super-resolution of less than 1 degree, and simultaneously perform short-range, medium-range, and long-range concurrent multi-mode sensing to quickly and reliably 4D imaging and breathing and heart rate monitoring are performed respectively for drivers and passengers; among them, behavior classification processing is performed through artificial intelligence models to realize the user's head degree of freedom, gestures, body posture, expressions, etc. that are perceived in 4D imaging. Parameter estimation enables application-oriented pattern recognition.

In some embodiments, the smart cockpit includes a DMS camera, and augmented reality data is obtained based on the fusion information data. Among them, DMS monitoring of the DMS camera and millimeter wave module requires joint calibration, that is, the internal and external parameters of the camera and the external parameters of the radar need to be registered. , and then form DMS combined data. In one embodiment, data such as vehicle body, ADAS, and positioning modules are added to perform multi-sensor information fusion calculations on the data layer, function layer, and feature layer, and finally effective AR information is extracted based on the fused information; wherein, effective AR information is extracted based on the fused information AR information includes multi-sensor information fusion processing based on the system's preset multi-sensor fusion algorithm to obtain effective AR information. Multi-sensor fusion algorithms include common multi-sensor fusion algorithms such as Kalman filtering and artificial neural networks.

In some embodiments, the rendering requirements include information display model rendering requirements or streaming media playback content or game rendering library requirements. The projection data obtained by rendering the augmented reality data according to the rendering requirement information includes, and the millimeter wave module performs enhancement processing according to the rendering requirements. The display model corresponding to the real data performs real-time rendering or loading of streaming media content, and the game rendering library renders the display model corresponding to the augmented reality data. Among them, augmented reality data is effective information based on AR.

In some embodiments, the fused information data and communication data are obtained through the millimeter wave module. The fused information data includes the monitoring information obtained by the millimeter wave module scanning the inside of the smart cockpit, and the communication data includes the Internet of Vehicles data obtained by the millimeter wave module inside and outside the smart cockpit. data, content delivery network data, and associated edge computing data. Augmented reality data is obtained based on the fusion of information data and communication data, which greatly expands the source of augmented reality data, making the augmented reality data not only come from the car cockpit (in addition to the communication and perception fusion information data between two local millimeter wave units, There are also local edge computing 3D models, rendering data, content distribution data, etc.), which also come from outside the car cockpit and extend to the Internet of Vehicles and the mobile Internet (remote rendering data, remote content distribution data from the mobile Internet, and widely distributed data from the Internet of Vehicles) augmented reality models, etc.). In one embodiment, the rendering requirement information is generated based on the augmented reality data, and the augmented reality data is rendered according to the rendering requirement information to obtain the projection data. There is no need to add an additional storage computing unit to the head-up display projector drive board of the head-up display projector. , the head-up display projector driver board can be dedicated to driving the optical part of the projector, thereby overcoming the limitations in some cases of the weak processing capabilities of the current vehicle-mounted augmented reality head-up display equipment, resulting in vehicle-mounted augmented reality heads-up displays. The display projector has a single function and can only provide users with some simple traffic information, making the augmented reality head-up display projector further provide users with driving information and entertainment content they need.

Referring to Figure 7, Figure 7 is a method flow chart of a smart cockpit control method proposed by another embodiment of the present application.

Embodiments of the present application provide a smart cockpit control method. The millimeter wave module includes a first millimeter wave unit and a second millimeter wave unit. The first millimeter wave unit is connected to the second millimeter wave unit, and the first millimeter wave unit is connected to the second millimeter wave unit. Projection device connection, methods include:

Step S710, control the first millimeter wave unit to obtain fusion information data, where the fusion information data includes the first monitoring information obtained by the first millimeter wave unit and the second monitoring information obtained by the second millimeter wave unit;

Step S720, control the first millimeter wave unit to obtain augmented reality data based on the fusion information data and the communication data obtained by the second millimeter wave unit;

Step S730, control the first millimeter wave unit to generate rendering requirement information based on the augmented reality data;

Step S740, control the first millimeter wave unit to send rendering requirement information to the second millimeter wave unit;

Step S750: Control the second millimeter wave unit to render the augmented reality data according to the rendering requirement information to obtain projection data, and send the projection data to the projection device for projection display.

In one embodiment, two millimeter wave module devices are provided inside the car cabin, namely a first millimeter wave unit and a second millimeter wave unit. The first millimeter wave unit is placed in the cockpit facing the driver and close to the AR HUD projector. The location is called the front millimeter wave module; the second millimeter wave unit is placed inside the top of the car cabin and is connected to the antenna array outside the roof, which is called the top millimeter wave module. While the two perform 4D imaging and breathing and heartbeat monitoring for drivers and passengers respectively, the first millimeter wave unit is also used as an AR generator algorithm processing center, and the second millimeter wave unit is also used as an edge computing center and content distribution network; both Cooperating with each other, on the one hand, it is used as an integral part of the car's in-cabin communication system and monitoring system for communication and perception; on the other hand, it is used to process the car's multi-sensor fusion information to generate effective AR information and perform AR rendering, as well as multimedia content and game playback. Rendering, the first millimeter wave unit obtains augmented reality data based on the fusion information data, and the first millimeter wave unit The meter-wave unit generates rendering demand information based on the augmented reality data. The first millimeter-wave unit sends rendering demand information to the second millimeter-wave unit. The rendering demand information carries augmented reality data. The second millimeter-wave unit processes the augmented reality data based on the rendering demand information. The rendering process obtains the projection data and sends the projection data to the head-up display projector for projection display.

In some embodiments, the first monitoring information is the driver's breathing, heartbeat, and movement information obtained by the first millimeter wave unit, and the second monitoring information is the passenger's breathing, heartbeat, and movement information obtained by the second millimeter wave unit. The physical signs and movement information of the driver and passengers are displayed on the head-up display projector, providing a safety warning function for the driver and passengers.

In some embodiments, the first millimeter wave unit has a millimeter wave antenna array that can be controlled by beamforming. On the one hand, it is used to monitor the driver's breathing and heartbeat (by detecting frequency modulated continuous wave signals with a specific range of values caused by tiny vibrations of the target). phase change) and 4D imaging of the driver's dangerous actions, distractions, and fatigue driving conditions (by judging the target distance, speed, horizontal angle, and pitch angle). On the other hand, the first millimeter wave unit has Stronger computing power replaces the traditional HUD driver board as the AR algorithm processing center.

In some embodiments, the second millimeter wave unit is connected to the roadside unit and/or the Internet of Vehicles cloud computing center, and the second millimeter wave unit renders the augmented reality data according to the rendering requirement information to obtain projection data including, the second millimeter wave unit The wave unit obtains updated data from the roadside unit and/or the Internet of Vehicles cloud computing center, performs augmented reality data rendering processing on the augmented reality data based on the updated data and demand information, and obtains projection data. Among them, the second millimeter wave unit is in the car cabin There are millimeter-wave antenna arrays that can be controlled by beamforming both inside and on the roof. On the one hand, they are used to monitor the breathing and heartbeat of passengers and 4D imaging monitoring of their behavior. On the other hand, they are used in the cabin edge computing center and The content distribution network provides powerful AR rendering computing power and streaming content distribution capabilities, and can communicate with the outside world through the antenna array on the roof of the car to the roadside unit, the Internet of Vehicles and the mobile Internet, and then update data and demand information based on Perform augmented reality data rendering processing on the augmented reality data to obtain projection data.

Referring to Figure 8, Figure 8 is a method flow chart of a smart cockpit control method proposed by another embodiment of the present application.

Embodiments of the present application provide a smart cockpit control method, wherein the millimeter wave module includes a first millimeter wave unit and a second millimeter wave unit, the first millimeter wave unit is connected to the second millimeter wave unit, and the first millimeter wave unit is connected to the second millimeter wave unit. Projection device connection, methods include:

Step S810, control the second millimeter wave unit to obtain fusion information data and communication data. The fusion information data includes the first monitoring information obtained by the first millimeter wave unit and the second monitoring information obtained by the second millimeter wave unit;

Step S820, control the second millimeter wave unit to obtain augmented reality data based on the fused information data and communication data;

Step S830, control the second millimeter wave unit to generate rendering requirement information based on the augmented reality data;

Step S840, control the second millimeter wave unit to render the augmented reality data according to the rendering requirement information to obtain projection data, and send the projection data to the projection device for projection display.

In one embodiment, the communication data includes Internet of Vehicles data acquired by the second millimeter wave unit, content analysis Send network data and associated edge computing data. When the first millimeter wave unit and the second millimeter wave unit work together, the AR information processing process becomes more complicated and requires higher computing power. During the playback process of the head-up display projector The first millimeter-wave unit is used to accurately calculate and load body perception elements such as the user's head degree of freedom, eyeballs, gestures, posture, and expressions into the AR generator. At the same time, the spatial positioning obtained by the user through AR glasses is also Accurately calculate and load it to the AR generator. At the same time, the first millimeter wave unit also serves as the AR generator. The calculation load and power consumption are very large. Therefore, through the large bandwidth, high capacity, and low time between the first millimeter wave unit and the second millimeter wave unit, Extended communication, the second millimeter wave unit obtains fusion information data, obtains augmented reality data based on the fusion information data, generates rendering demand information based on the augmented reality data, renders the augmented reality data based on the rendering demand information to obtain projection data, and projects the The data is sent to the head-up display projector for projection display, and then the vehicle head-up display projector can provide users with augmented reality services more quickly and stably, and display driving information and entertainment content to users according to their needs.

Referring to Figure 9, Figure 9 is a method flow chart of an intelligent cockpit control method proposed by another embodiment of the present application.

Step S910, control the first millimeter wave unit and the second millimeter wave unit to perform rendering task allocation processing according to their respective computing power consumption data, and obtain the allocation result;

Step S920: Control the first millimeter wave unit and the second millimeter wave unit to process the fusion information data and communication data according to the allocation results to obtain augmented reality data. The fusion information data includes the first monitoring information and the third millimeter wave unit obtained by the first millimeter wave unit. The second monitoring information obtained by the two millimeter wave unit;

Step S930, control the second millimeter wave unit to generate rendering requirement information based on the augmented reality data;

Step S940, control the second millimeter wave unit to render the augmented reality data according to the rendering requirement information to obtain projection data, and send the projection data to the projection device for projection display.

In one embodiment, the computing power consumption data represents the processing capabilities of the first millimeter wave unit and the second millimeter wave unit, and the first millimeter wave unit and the second millimeter wave unit perform task allocation processing according to their respective computing power consumption data. , to obtain the allocation result, the first millimeter wave unit and the second millimeter wave unit collaboratively process the fusion information data and communication data according to the allocation result, that is, federated computing processing, to obtain augmented reality data. The fusion information data includes the information obtained by the first millimeter wave unit. The first monitoring information and the second monitoring information obtained by the second millimeter wave unit, wherein the federal calculation is to allocate computing tasks to the first millimeter wave unit and the second millimeter wave unit, so that the first millimeter wave unit and the second millimeter wave unit The wave unit can allocate loads, share data files and memory based on the allocation results, perform various assigned computing tasks, and collaboratively process subsequent rendering processing tasks.

In one embodiment, the communication data includes Internet of Vehicles data, content distribution network data and associated edge computing data acquired by the first millimeter wave unit and/or the second millimeter wave unit. Unit division of labor makes the AR information processing process more complex and requires higher computing power. During the playback process of the head-up display projector, the first millimeter wave unit is used to combine the user's head degree of freedom, eyeballs, gestures, and body posture. , expressions and other body perception elements are accurately calculated and loaded into the AR generator, while also The spatial positioning obtained by the user through AR glasses must be accurately calculated and loaded into the AR generator. At the same time, the first millimeter wave unit also serves as the AR generator. The calculation load and power consumption are very large. Therefore, through the first millimeter wave unit and the second For large-bandwidth, high-capacity, and low-latency communication between millimeter-wave units, the second millimeter-wave unit participates in part or all of the multi-sensor sensing calculations, that is, federated computing is introduced, and the first millimeter-wave unit and the second millimeter-wave unit perform Federal calculation is performed to obtain the federal calculation result. The first millimeter wave unit and the second millimeter wave unit process the fusion information data according to the federation calculation result to obtain augmented reality data. The fusion information data includes the first monitoring information obtained by the first millimeter wave unit. and the second monitoring information obtained by the second millimeter wave unit to achieve the balancing of computing power and power consumption between the two millimeter wave modules. The first millimeter wave unit and the second millimeter wave unit can achieve computing power balancing through mutual communication to ensure that the unit The user's body perception and spatial positioning of the universe are well integrated into the interactive virtual and real XR space, achieving a perfect integration of the virtual and real worlds.

Referring to Figure 10, an embodiment of the present application also provides a smart cockpit controller 1000, which includes: a memory 1020, a processor 1010, and a computer program stored in the memory and executable on the processor. When the processor 1010 executes the computer program Implement the smart cockpit control method as in any one of the above embodiments, for example, perform the above-described method steps S610 to S650 in Figure 6, method steps S710 to S750 in Figure 7, and method steps S810 to S840 in Figure 8 , method steps S910 to S940 in Figure 9.

In addition, the embodiment of the present application also provides a smart cockpit, which includes the above-mentioned controller. Since the smart cockpit of the embodiment of the present application has the controller of the above-mentioned embodiment, and the controller of the above-mentioned embodiment can execute The smart cockpit control method in the above embodiments, therefore, the implementation manner and technical effects of the smart cockpit in the embodiment of the present application can be referred to the implementation manner and technical effects of the smart cockpit control method in any of the above embodiments.

In addition, an embodiment of the present application also provides a computer-readable storage medium that stores computer-executable instructions, and the computer-executable instructions are executed by one or more control processors, for example, execute The method steps S610 to S650 in FIG. 6 , the method steps S710 to S750 in FIG. 7 , the method steps S810 to S840 in FIG. 8 , and the method steps S910 to S940 in FIG. 9 are described above.

This application has at least the following beneficial effects: through the intelligent cockpit control method proposed in this application, fusion information data and communication data are obtained. The fusion information data includes monitoring information obtained by scanning the interior of the intelligent cockpit by the millimeter wave module. The communication data includes the millimeter wave module and The communication data that interacts inside and outside the smart cockpit is used to obtain augmented reality data based on the fusion of information data and communication data. The augmented reality data is rendered and processed to obtain projection data, and the projection data is sent to the projection device for projection display. Using the millimeter wave module as the augmented reality generator, it obtains augmented reality data based on multi-sensor fusion data and communication data, renders the augmented reality data, and provides users with augmented reality and elements more quickly and stably through the vehicle head-up display projector. Universe services, and then display driving information and entertainment content to users according to their needs.

Those of ordinary skill in the art can understand that all or some steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a processor, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software can Distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. removable, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical disk storage, magnetic cassettes, tapes, disk storage or other magnetic storage devices, or may Any other medium used to store the desired information and that can be accessed by a computer. Additionally, it is known to those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Claims

An intelligent cockpit control method, applied to an intelligent cockpit. The intelligent cockpit includes a millimeter wave module, and the millimeter wave module is connected to a projection device. The method includes:

Obtain fusion information data and communication data, the fusion information data includes monitoring information obtained by scanning the interior of the smart cockpit by the millimeter wave module, and the communication data includes communication data between the millimeter wave module and the inside and outside of the smart cockpit;

Obtain augmented reality data according to the fused information data and the communication data;

Perform rendering processing on the augmented reality data to obtain projection data;

The projection data is sent to the projection device for projection display.
The intelligent cockpit control method according to claim 1, wherein the communication data includes at least one of the following: networking data, content distribution network data or associated edge computing data.
The intelligent cockpit control method according to claim 1, wherein the millimeter wave module includes a first millimeter wave unit and a second millimeter wave unit, and the method includes:

Control the first millimeter wave unit to obtain the fusion information data, where the fusion information data includes the first monitoring information obtained by the first millimeter wave unit and the second monitoring information obtained by the second millimeter wave unit;

Control the first millimeter wave unit to obtain augmented reality data based on the fusion information data and the communication data acquired by the second millimeter wave unit;

Control the first millimeter wave unit to generate rendering requirement information according to the augmented reality data;

Control the first millimeter wave unit to send rendering requirement information to the second millimeter wave unit;

The second millimeter wave unit is controlled to render the augmented reality data according to the rendering requirement information to obtain projection data, and send the projection data to the projection device for projection display.
The intelligent cockpit control method according to claim 1, wherein the millimeter wave module includes a first millimeter wave unit and a second millimeter wave unit, and the method includes:

Control the first millimeter wave unit and the second millimeter wave unit to perform rendering task allocation processing according to their respective computing power consumption data to obtain allocation results;

Control the first millimeter wave unit and the second millimeter wave unit to process the fusion information data and the communication data according to the allocation result to obtain augmented reality data, where the fusion information data includes the first The first monitoring information obtained by the millimeter wave unit and the second monitoring information obtained by the second millimeter wave unit;

Control the second millimeter wave unit to generate rendering requirement information according to the augmented reality data;

The second millimeter wave unit is controlled to render the augmented reality data according to the rendering requirement information to obtain projection data, and send the projection data to the projection device for projection display.
The intelligent cockpit control method according to any one of claims 1 to 4, wherein the fusion information data also includes sensory information, and obtaining augmented reality data based on the fusion information data includes:

Augmented reality data is obtained according to the monitoring information and the sensing information.
The intelligent cockpit control method according to claim 5, wherein the sensing information includes At least one of the following: user eye tracking data, user keystroke data or spatial positioning data.
The intelligent cockpit control method according to any one of claims 1 to 4, wherein before obtaining the fusion information data, it further includes:

Scan the cabin where the millimeter wave module is located to obtain the positioning calculation value;

Obtain target location information according to the positioning calculation value;

The monitoring information is obtained according to the target location information.
The smart cockpit control method according to any one of claims 1 to 4, wherein the millimeter wave module is provided with a millimeter wave antenna array, and before obtaining the fusion information data, it further includes:

Obtain the phase interference difference value of the millimeter wave antenna array;

Obtain target position information according to the phase interference difference value;

The monitoring information is obtained according to the target location information.
The smart cockpit control method according to claim 3 or 4, wherein the second millimeter wave unit is connected to the roadside unit and/or the Internet of Vehicles cloud computing center, the second millimeter wave unit is connected to the mobile Internet, and the control The second millimeter wave unit performs rendering processing on the augmented reality data according to the rendering requirement information to obtain projection data including:

Control the second millimeter wave unit to obtain update data from the roadside unit and/or the Internet of Vehicles cloud computing center, and content distribution network data from the mobile Internet;

The second millimeter wave unit is controlled to perform augmented reality data rendering processing on the augmented reality data according to the update data, the content distribution network data and the demand information to obtain projection data.
An intelligent cockpit controller includes a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements claims 1 to 9 The intelligent cockpit control method described in any one of the above.
An intelligent cockpit, characterized by comprising the controller according to claim 10.
A computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the intelligent cockpit control method according to any one of claims 1 to 9.