WO2023216580A1 - 调节显示设备的方法和装置 - Google Patents

调节显示设备的方法和装置 Download PDF

Info

Publication number
WO2023216580A1
WO2023216580A1 PCT/CN2022/137563 CN2022137563W WO2023216580A1 WO 2023216580 A1 WO2023216580 A1 WO 2023216580A1 CN 2022137563 W CN2022137563 W CN 2022137563W WO 2023216580 A1 WO2023216580 A1 WO 2023216580A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
user
display device
facial
orientation
Prior art date
Application number
PCT/CN2022/137563
Other languages
English (en)
French (fr)
Inventor
袁麓
查敬芳
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023216580A1 publication Critical patent/WO2023216580A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1415—Digital output to display device ; Cooperation and interconnection of the display device with other functional units with means for detecting differences between the image stored in the host and the images displayed on the displays

Definitions

  • the present application relates to the field of human-computer interaction, and more specifically, to a method and device for adjusting a display device.
  • imaging display devices are integrated into the cockpit, such as large vehicle screens, head up displays (HUD), vehicle holographic images and other display devices.
  • HUD head up displays
  • the display range of existing display devices is limited. When the user is in different positions, these display devices are not necessarily in a better display angle or position for the user, and the imaging may even be incomplete and the display is invisible. .
  • This application provides a method and device for adjusting a display device, which can accurately adjust the orientation of the image displayed by the display device according to the user's position, helping to improve the user experience at the level of human-computer interaction.
  • the first aspect provides a method for adjusting a display device, which method can be executed by a vehicle; or it can also be executed by a vehicle-mounted terminal such as a vehicle machine; or it can also be executed by a chip or circuit for the vehicle, This application does not limit this.
  • the following takes vehicle execution as an example.
  • the method may include: obtaining a first facial image of the first user; determining first spatial coordinates of the first user's facial feature points based on the first facial image; adjusting the image displayed by the display device based on the first spatial coordinates. The orientation of the first side of the image that contains the information to be conveyed to the first user.
  • the orientation of the image displayed by the display device is accurately adjusted according to the user's position, which can improve the user experience at the interactive level.
  • the above-mentioned first spatial coordinates are the three-dimensional coordinates of the user's facial feature points in the vehicle coordinate system.
  • the facial feature points can be located at any one or more of the first user's eyebrows, eyes, eyebrows, and nose; or, the facial feature points can also be points at other locations on the first user's face. This application The examples do not specifically limit this.
  • the above image can be a three-dimensional image, such as holographic projection. It should be understood that different angles of holographic projection contain different information.
  • the above image may also be a two-dimensional image with three-dimensional features that interacts with the first user, such as a digital human. It should be understood that different interaction effects can be achieved when the digital person's face faces in different directions. For example, when the digital person's face faces the first user, the first user can feel the effect of face-to-face communication with the digital person.
  • the first side of the image may be the side containing the front face of the virtual assistant.
  • the image is a three-dimensional image of the own car.
  • the first side of the image can be the side containing the vehicle cowl.
  • adjusting the orientation of the first side of the image may include adjusting the posture of the image.
  • the head posture of the image may be adjusted so that its face faces the first user.
  • the method further includes: after adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinate, obtaining the second face of the second user. Image; determine second spatial coordinates of the facial feature points of the second user based on the second facial image; adjust the orientation of the first side of the image displayed by the display device based on the second spatial coordinates.
  • the determination of the second spatial coordinates may be actively triggered by the second user.
  • the second user triggers through a voice command, for example, the second user issues a voice command "holographic image, please face me”; or the second user triggers through a relevant button, for example, the second user clicks a button that "turns on the holographic image”.
  • the vehicle control camera device acquires a second facial image of the second user, and then determines the second spatial coordinates of the facial feature points of the second user.
  • a second facial image of the second user is acquired outside a preset period of time after adjusting the orientation of the first side of the image displayed on the display device according to the first spatial coordinate.
  • the above-mentioned preset time length may be 3 seconds, or 5 seconds, or may be other preset time lengths, which are not specifically limited in this embodiment of the present application.
  • the orientation of the image displayed by the display device can be adjusted according to the three-dimensional coordinates of the facial feature points of each user among the multiple users, thereby improving the performance of the multi-user scenario. interactive experience.
  • the method further includes: after adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinate, obtaining a third image of the first user. Facial image; determine the third spatial coordinates of the first user's facial feature points based on the third facial image; when the distance between the first spatial coordinates and the third spatial coordinates is greater than or equal to a preset threshold, based on The third spatial coordinate adjusts the orientation of the first side of the image displayed by the display device.
  • the head position of the first user changes.
  • the second facial image of the first user can be obtained, and the person can be determined based on the second facial image.
  • the second coordinate position of the facial feature point when the first user interacts with the image, the head position of the first user changes.
  • the orientation of the first side of the image is adjusted according to the second spatial coordinates.
  • the preset threshold may be 15 centimeters, or 20 centimeters, or other distances, which are not specifically limited in the embodiments of the present application.
  • the method further includes: before acquiring the first facial image of the first user, acquiring audio information, where the audio information includes the first user's voice instruction, The voice instruction is used to instruct turning on and/or adjusting the display device; determining the sound source position of the audio information; and determining the first user's position based on the sound source position.
  • the position of the sound source can be used to determine whether the user issuing the command is the driver, co-driver or rear passenger, and then adjust the corresponding display device so that the display The side of the image displayed by the device that contains the information intended to be conveyed to the user faces the user, thereby improving the interactivity between the user and the image displayed by the display device.
  • the audio information also includes a second user's voice instruction
  • the method further includes: controlling the display device to display the first image and the second image; adjusting the display device Make the first side of the first image face the first user, and the first side of the second image face the second user, wherein the image displayed by the display device includes the first image and/or the second image , the first side of the first image contains information to be conveyed to the first user, and the first side of the second image contains information to be conveyed to the second user.
  • the display device can also copy images, so that each image faces one of the multiple users, further improving the user's interactive experience and driving fun.
  • the method further includes: adjusting the position of the image displayed by the display device according to the first spatial coordinate.
  • the position of the image in the y-axis direction of the vehicle coordinate system can be adjusted according to the first spatial coordinate. For example, after determining the user's position, the display device can be adjusted so that the image projected by the display device Move to a location that corresponds to the user's location. For example, if the user's position is the passenger's seat, the display device can be adjusted so that the position of the image moves to the passenger's seat.
  • the position of the image in the z-axis direction of the vehicle coordinate system can be adjusted according to the first spatial coordinate. For example, after determining the user's position, the display device can be adjusted so that the image projected by the display device Move the height to a height suitable for user viewing.
  • the y-axis direction in the vehicle coordinate system is a direction perpendicular to the vehicle's longitudinal symmetry plane; the z-axis direction in the vehicle coordinate system is a direction parallel to the vehicle's longitudinal symmetry plane.
  • the display device is adjusted to control the image to move to the user's position, or to control the image to move to an appropriate height so that the user does not have to turn his head to interact with the image displayed by the display device, which helps to improve the user's comfort level. Driving experience.
  • the method further includes: before adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinate, obtaining an initial pose of the display device.
  • Information the initial pose information is used to indicate the attitude angle and position of the display device;
  • adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinates includes: according to the first spatial coordinates and the initial The pose information determines the rotation angle and rotation direction of the display device; the orientation of the first side of the image displayed by the display device is adjusted according to the rotation angle and the rotation direction.
  • the initial pose information of the image is obtained, and the initial pose information of the image is used to indicate the image.
  • the attitude angle and position of the image are determined; the rotation angle and rotation direction of the image are determined based on the first spatial coordinates and the initial pose information of the image. Further, according to the mapping relationship between the angle transformation of the display device and the angle transformation of the image, the rotation angle and rotation direction of the display device are determined, and then the display device is adjusted. For example, if the above image is a virtual assistant, the posture angle of the image may be the head posture angle of the virtual assistant.
  • the first spatial coordinates of the first user's facial feature points are used to indicate the three-dimensional position of the first user's facial feature points in the vehicle, and the The vehicle includes the display device.
  • the display device can be adjusted according to the three-dimensional coordinates of the user's facial feature points in the vehicle coordinate system, so that the orientation of the image displayed by the display device can be accurately adjusted to provide users with better interactive effects. Prompt the user's interactive experience.
  • the display device includes a holographic projection device, and the image includes a three-dimensional 3D image.
  • a device for adjusting a display device includes: an acquisition unit, configured to acquire a first facial image of a first user; and a processing unit, configured to determine the first facial image based on the first facial image. First spatial coordinates of the user's facial feature points; adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinates.
  • the acquisition unit is further configured to: after adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinate, acquire the second user's third Two facial images; the processing unit is also configured to: determine the second spatial coordinates of the facial feature points of the second user based on the second facial image; adjust the first coordinate of the image displayed by the display device based on the second spatial coordinates side orientation.
  • the acquisition unit is further configured to: after adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinate, acquire the first user's orientation a third facial image; the processing unit is further configured to: determine the third spatial coordinates of the first user's facial feature points based on the third facial image; the distance between the first spatial coordinates and the third spatial coordinates When it is greater than or equal to the preset threshold, the orientation of the first side of the image displayed by the display device is adjusted according to the third spatial coordinate.
  • the acquisition unit is further configured to: before acquiring the first facial image of the first user, acquire audio information, where the audio information includes the first user's first facial image.
  • Voice instructions the voice instructions are used to instruct turning on and/or adjusting the display device; the processing unit is also used to: determine the sound source position of the audio information; determine the position of the first user based on the sound source position.
  • the audio information also includes a voice command of the second user
  • the processing unit is further configured to: control the display device to display the first image and the second image; adjust the The display device causes the first side of the first image to face the first user, and the first side of the second image to face the second user, wherein the image displayed by the display device includes the first image and/or the second image.
  • Two images, the first side of the first image contains information to be conveyed to the first user, and the first side of the second image contains information to be conveyed to the second user.
  • the processing unit is further configured to: adjust the position of the image displayed by the display device according to the first spatial coordinates.
  • the acquisition unit is further configured to: acquire an initial orientation of the display device before adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinate.
  • Pose information the initial pose information is used to indicate the attitude angle and position of the display device;
  • the processing unit is also used to: determine the rotation angle and rotation direction of the display device based on the first spatial coordinates and the initial pose information ; According to the rotation angle and the rotation direction, adjust the orientation of the first side of the image displayed by the display device.
  • the first spatial coordinates of the first user's facial feature points are used to indicate the three-dimensional position of the first user's facial feature points in the vehicle, and the The vehicle includes the display device.
  • the display device includes a holographic projection device, and the image includes a three-dimensional 3D image.
  • a device for adjusting a display device includes: a memory for storing a program; a processor for executing the program stored in the memory.
  • the processor is configured to execute the above A method in any possible implementation manner of the first aspect.
  • a fourth aspect provides a vehicle, which includes the device in any possible implementation of the second aspect or the device in any implementation of the third aspect, and the above display device.
  • the display device can be a holographic projection device, and the displayed image can be a three-dimensional image; or the display device can also be a vehicle display screen, and the displayed image can be a digital human; or the display device can also be other display devices. , this application does not specifically limit this.
  • a computer program product includes: computer program code.
  • the computer program code When the computer program code is run on a computer, it causes the computer to execute the method in any of the possible implementation modes of the first aspect. .
  • the above computer program code may be stored in whole or in part on the first storage medium, where the first storage medium may be packaged together with the processor, or may be packaged separately from the processor. This is not the case in the embodiments of this application. Specific limitations.
  • a computer-readable medium stores program code.
  • the computer program code When the computer program code is run on a computer, it causes the computer to execute any of the possible implementation methods of the first aspect. method.
  • a seventh aspect provides a chip, which includes a processor for calling a computer program or computer instructions stored in a memory, so that the processor executes the method in any of the possible implementations of the first aspect.
  • the processor is coupled with the memory through an interface.
  • the chip system further includes a memory, and a computer program or computer instructions are stored in the memory.
  • a method for adjusting a display device can accurately adjust the orientation of the image displayed by the display device according to the user's position, thereby improving the user's experience at the interactive level.
  • the display device can be adjusted according to the position of the user's head movement. When the user's head position moves beyond the preset distance, the orientation of the image is adjusted according to the position after the movement. This can avoid frequent adjustments to the orientation of the image and help improve User’s driving experience.
  • the orientation of the image displayed on the display device can be adjusted based on the three-dimensional coordinates of the facial feature points of each user among the multiple users, thereby improving the interactive experience in a multi-user scenario.
  • the position of the sound source can be used to determine whether the user issuing the command is the driver, co-driver or rear passenger, and then adjust the corresponding display device so that the image displayed by the display device is included.
  • the side where the information is intended to be conveyed to the user faces the user, thereby improving the interactivity between the user and the image displayed on the display device.
  • the display device can also copy images, so that each image faces one of multiple users, further improving the user's interactive experience and driving fun.
  • FIG. 1 is a schematic diagram of an application scenario of a method for adjusting a display device provided by an embodiment of the present application.
  • FIG. 2 is a schematic block diagram of a vehicle 100 provided by an embodiment of the present application.
  • FIG. 3 is a schematic block diagram of a system for adjusting a display device provided by an embodiment of the present application.
  • FIG. 4 is a schematic flow chart of a method for adjusting a display device provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of the transformation relationship between the image coordinate system and the camera coordinate system provided by the embodiment of the present application.
  • FIG. 6 is a schematic diagram of an application scenario for adjusting images displayed by a display device according to an embodiment of the present application.
  • FIG. 7 is another schematic flowchart of a method for adjusting a display device provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of another application scenario for adjusting the image displayed by the display device according to the embodiment of the present application.
  • FIG. 9 is yet another schematic flowchart of a method for adjusting a display device provided by an embodiment of the present application.
  • FIG. 10 is a schematic block diagram of a device for adjusting a display device provided by an embodiment of the present application.
  • FIG. 11 is a schematic block diagram of a device for adjusting a display device provided by an embodiment of the present application.
  • Figure 1 is a schematic diagram of a vehicle cockpit scene provided by an embodiment of the present application.
  • One or more cameras can be installed inside the smart cockpit to capture images inside or outside the cabin, such as the camera of the driver monitor system (DMS), the camera of the cabin monitor system (CMS) Cameras, and dashcam cameras.
  • DMS driver monitor system
  • CMS cabin monitor system
  • dashcam cameras the camera installed on the A-pillar as an example.
  • the cameras used to capture the interior and exterior of the cabin can be the same camera or different cameras.
  • a vehicle-mounted display screen is also provided in the cockpit.
  • (a) in Figure 1 takes the display screen provided in the central control area as an example.
  • the vehicle can display information to the user through at least one of a vehicle-mounted display screen, a HUD (not shown in Figure 1(a)), and a vehicle-mounted holographic projection device (not shown in Figure 1(a)).
  • a vehicle-mounted display screen a HUD (not shown in Figure 1(a))
  • a vehicle-mounted holographic projection device not shown in Figure 1(a)
  • the three-dimensional image of the virtual assistant can be displayed to the user through a vehicle-mounted holographic projection device, as shown in (a) in Figure 1 .
  • the vehicle-mounted holographic projection device can also perform imaging on the back of the front seat headrest, as shown in (b) in Figure 1.
  • the three-dimensional image shown in (a) in Figure 1 is only an exemplary illustration, and the specific content of the three-dimensional image can also be other three-dimensional objects, for example, it can also be a three-dimensional image of the own car. It should be understood that in the embodiment of the present application, the position of the camera that collects image information in the cockpit is not specifically limited.
  • the camera can be located on the A-pillar as shown in (a) in Figure 1, or on the B-pillar, or under the steering wheel, or near the rearview mirror.
  • FIG. 2 is a functional block diagram of the vehicle 100 provided by the embodiment of the present application.
  • Vehicle 100 may include perception system 120, display device 130, and computing platform 150.
  • the sensing system 120 may include several types of sensors that sense information about the environment surrounding the vehicle 100 as well as the environment inside the vehicle cabin.
  • the sensing system 120 may include a positioning system.
  • the positioning system may be a global positioning system (GPS), a Beidou system or other positioning systems, an inertial measurement unit (IMU), a lidar, a millimeter One or more of wave radar, ultrasonic radar, visual sensor, sound sensor and camera device.
  • GPS global positioning system
  • IMU inertial measurement unit
  • lidar a millimeter One or more of wave radar, ultrasonic radar, visual sensor, sound sensor and camera device.
  • the above-mentioned camera device can include a red, green and blue/infrared (RGB/IR) camera, or a depth camera, such as a time of flight (TOF) camera, a binocular camera, or a structured light camera. wait.
  • RGB/IR red, green and blue/infrared
  • TOF time of flight
  • binocular camera a binocular camera
  • structured light camera a structured light camera
  • the computing platform 150 may include processors 151 to 15n (n is a positive integer).
  • the processor is a circuit with signal processing capabilities.
  • the processor may be a circuit with instruction reading and execution capabilities.
  • CPU central processing unit
  • microprocessor microprocessor
  • GPU graphics processing unit
  • DSP digital signal processor
  • the processor can realize certain functions through the logical relationship of the hardware circuit. The logical relationship of the hardware circuit is fixed or can be reconstructed.
  • the processor is an application-specific integrated circuit (application-specific integrated circuit).
  • Hardware circuits implemented by ASIC or programmable logic device (PLD), such as field programmable gate array (FPGA).
  • PLD programmable logic device
  • FPGA field programmable gate array
  • the process of the processor loading the configuration file and realizing the hardware circuit configuration can be understood as the process of the processor loading instructions to realize the functions of some or all of the above units.
  • it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing Unit (deep learning processing unit, DPU), etc.
  • the computing platform 150 may also include a memory, which is used to store instructions. Some or all of the processors 151 to 15n may call instructions in the memory and execute the instructions to implement corresponding functions.
  • the display device 130 in the cockpit is mainly divided into three categories.
  • the first category is a vehicle-mounted display screen;
  • the second category is a projection display screen, such as a HUD;
  • the third category is a holographic projection device.
  • the vehicle display screen is a physical display screen and an important part of the vehicle infotainment system.
  • Head-up display also known as head-up display system, includes combined head-up display (combiner-HUD, C-HUD) system, windshield-type head-up display (windshield-HUD, W-HUD) system, augmented reality head-up display system (augmented reality HUD) , AR-HUD).
  • the panoramic imaging system is a display device that can display three-dimensional (3D) images of objects.
  • the holographic projection device can record and reproduce the 3D image of the object based on the principles of light interference and diffraction, or it can also achieve 3D effects through edge blanking, Pepper’s ghost and other methods.
  • the image of the 3D virtual assistant, the 3D image of the own car, or other 3D images can be displayed through the holographic projection device, which is not specifically limited in the embodiment of the present application.
  • the display device 130 can also display a digital human. It should be understood that a digital human can refer to a character with human appearance, personality and other characteristics, the ability to express language, facial expressions and body movements, and the ability to communicate with humans. Communicate and interact with virtual images, which can be displayed on vehicle displays, etc.
  • the image displayed by the HUD is a two-dimensional image projected on the HUD screen; the digital person displayed on the vehicle display screen is also included in the two-dimensional image with three-dimensional effect and depth of field; and the image displayed by the holographic projection device is three-dimensional.
  • Three-dimensional images that is, observing holographic images from different angles, you can see different sides of the object, with parallax effects and a sense of depth of field.
  • the processor can obtain the user image information collected by the perception system 120, and determine the user's eye position and the user's line of sight focus based on the three-dimensional coordinate system of the vehicle 100. After determining the user's line of sight focus point on the display device and/or on the image displayed by the display device, the display device is adjusted according to the user's actual position in the vehicle 100 .
  • the processor can also determine the user's location based on the sound information collected by the sound sensor in the perception system 120, and then adjust the angle of the image projected by the holographic projection device according to the user's location.
  • the above user image information and sound information can also be stored in the memory in the computing platform 150 in the form of data.
  • the processor can process the above user image information and sound information to obtain parameterized indicators, and then adjust the display device according to the parameterized indicators. It should be understood that the above operations can be performed by the same processor, or can also be performed by one or more processors, which is not specifically limited in the embodiments of the present application.
  • Figure 3 shows a system architecture diagram for adjusting a display device provided by an embodiment of the present application.
  • the system 200 includes a sensing module, a calibration module, a display device adjustment module and a display module.
  • the sensing module may include one or more camera devices in the sensing system 120 shown in Figure 2, and one or more sensors; the display module may include one or more of the display devices 130 shown in Figure 2.
  • Multiple display devices; calibration modules and display device adjustment modules may include one or more processors in computing platform 150 shown in FIG. 2 .
  • the perception module can detect the user's facial information and/or voice information, and the display device adjustment module can determine the adjustment parameters based on the face image information obtained from the perception module and the external parameters in the vehicle coordinate system obtained from the calibration module, Then, the actual three-dimensional coordinates of any position of the user's face in the vehicle are determined based on the external parameters in the vehicle coordinate system and the face image information. Further, the display device adjustment module can adjust the display device in the display module according to the three-dimensional coordinates of one or several feature points of the user's face. For example, a certain feature point of the user's face may be the eyes, or it may be a point between the two eyes, or it may be other points, which is not specifically limited in the embodiment of the present application.
  • the above-mentioned modules and devices are only examples. In actual applications, the above-mentioned modules and devices may be added or deleted according to actual needs.
  • the display module and the display device adjustment module in Figure 3 can also be combined into one module, that is, the functions of the two are implemented by one module.
  • the display range of display devices under the current technical background is limited.
  • these display devices are not necessarily in a better display angle or position for the user, and may even cause incomplete imaging, Show invisible situations.
  • the direction of the three-dimensional image projected by it is fixed, and users at different locations see different images, which may cause users who want to obtain the information conveyed by the image to be unable to obtain relevant information in a timely and comprehensive manner.
  • embodiments of the present application provide a method and device for adjusting a display device, which can adjust the holographic projection device according to the user's position, so that the three-dimensional image projected by it can be accurately adjusted according to the user's position, thereby enhancing communication with the user. interactivity, thereby improving the user’s driving experience.
  • Figure 4 shows a method for adjusting a display device provided by an embodiment of the present application.
  • the method 400 can be applied to the scenario shown in Figure 1 or to the vehicle 100 shown in Figure 2. This method can also be used It is executed by the system shown in Figure 3.
  • the method 400 includes:
  • the user's first facial image can be obtained through a camera installed inside the vehicle, where the first facial image includes the user's eye image.
  • the camera can be set on the A-pillar so that the camera can capture the user's eyes.
  • the user can be any user in the vehicle cabin, for example, the user can be the driver, the user in the passenger seat, or the user in the rear seat of the cabin.
  • S402 Determine whether the user's focus is on the displayed image.
  • S403 when it is determined that the user's line of sight is focused on the displayed image, S403 is executed; otherwise, S405 is executed.
  • the display device is a holographic projection device, and the image projected by it is a virtual 3D image. Then it is determined whether the user's focus is on the virtual 3D image.
  • the user's gaze focus point may be determined through gaze estimation technology or gaze tracking technology.
  • the first position information is used to indicate the first position of the facial feature point in the vehicle coordinate system.
  • one facial feature point can be determined based on the first facial image, or two or more facial feature points can be determined based on the first facial image. This is not specifically limited in the embodiments of the present application. .
  • the facial feature points can be located at any one or more of the center of the eyebrows, eyes, eyebrows, and nose; or the facial feature points can also be points at other locations on the user's face, which is not specified in the embodiments of this application. limited.
  • the three-dimensional coordinates of the facial feature point inside the vehicle cabin are determined based on the image coordinates of the facial feature point on the first facial image.
  • the facial feature area may also be determined based on the first facial image, and then the first location information may be determined based on the facial feature area.
  • the first location information may also be determined through other feature points or areas determined in the first facial image, which is not specifically limited in this application.
  • the image coordinate system is a coordinate system established on the image captured by the camera.
  • the image coordinate system may take the center of the image captured by the camera as the origin, and the above image coordinate system is a two-dimensional coordinate system.
  • the camera coordinate system takes the optical center of the camera as the origin of the coordinate system.
  • the Xc and Yc axes are parallel to the x and y axes of the image coordinate system.
  • the optical axis of the camera is the Zc axis.
  • the coordinate system satisfies the right-hand rule.
  • the optical center of the camera can be understood as the geometric center of the camera lens, and the above-mentioned camera coordinate system is a three-dimensional coordinate system.
  • the transformation relationship between the image coordinate system and the camera coordinate system can be shown in Figure 5.
  • the two-dimensional coordinate system o-xy is the image coordinate system of the image q captured by the camera.
  • the intersection of the x-axis and the y-axis is o, that is, o is the origin of the image coordinate system, and the coordinates of any point on the image plane q can be expressed by image coordinates (x, y).
  • the three-dimensional coordinate system O-XYZ is the camera coordinate system, and any plane in the camera coordinate system can be called a space plane.
  • the space plane Q is any plane in the space coordinate system (that is, the space plane Q is any plane in O-XYZ), and the coordinates of any point on the space plane Q can be expressed by three-dimensional coordinates (X, Y, Z).
  • q represents the image coordinates of the point in the image plane
  • Q represents the three-dimensional coordinates of the point in the space plane
  • s is the scale scale factor
  • H MW.
  • M is the internal parameter matrix of the camera
  • f x represents the product of the physical focal length length of the lens in the camera and the size of each imaging unit in the x direction
  • f y represents the product of the physical focal length length of the lens in the camera and the size of each imaging unit in the y direction
  • c x represents the offset of the projection coordinate center relative to the optical axis in the x direction
  • c y represents the offset of the projection coordinate center relative to the optical axis in the y direction.
  • Each unit size of the image can be one pixel in the image captured by the camera.
  • W [R, t]
  • W is the external parameter matrix of the camera.
  • R is a 3*3 orthogonal unit matrix, also called a rotation matrix
  • t is a three-dimensional translation vector. If the image plane q and the space plane Q have the above transformation relationship, and the image coordinates, scale scale factor and homography matrix H of a certain point on q are known, the point on Q can be calculated based on the above transformation relationship between the image plane and the space plane. spatial coordinates.
  • the image coordinates of a certain point on plane q are (x 1 , y 1 ). If the internal parameter matrix of the camera is M, the external parameter matrix of the camera is W, and the scale scale factor is s, then according to the image plane and From the transformation relationship of the space plane, we can get the equation with X 1 , Y 1 , and Z 1 as unknown variables:
  • the coordinates (X 1 , Y 1 , Z 1 ) of the point on the space plane Q are converted into the vehicle coordinate system to obtain the three-dimensional coordinates (X 2 , Y 2 , Z) of the point in the vehicle coordinate system. 2 ).
  • the three-dimensional coordinates (X 2 , Y 2 , Z 2 ) can be determined through a perspective-n-point (PnP) algorithm based on a priori head model.
  • the prior head model includes the relationship between the vehicle coordinate system and the image coordinate system of the specified point. Specifically, there is the following relationship between the coordinates of a certain point in the vehicle coordinate system and the coordinates of the point in the camera coordinate system:
  • R CW is the pose transformation from the vehicle coordinate system to the camera coordinate system. More specifically, R CW is the rotation matrix from the vehicle coordinate system to the camera coordinate system (converting the representation of the same vector in the vehicle coordinate system to the camera coordinate system representation in the coordinate system), is the corresponding translation vector (that is, the vector from the origin of the camera coordinate system to the origin of the vehicle coordinate system, expressed in the camera coordinate system).
  • R CW based on the coordinates of n points in the a priori head model in the vehicle coordinate system and the coordinates of these n points in the image coordinate system, R CW and Further, according to the coordinates (X 1 , Y 1 , Z 1 ) of the facial feature point in the camera coordinate system, the three-dimensional coordinates (X 2 , Y 2 , Z 2 ) of the point in the vehicle coordinate system can be obtained.
  • the 3D position of the corresponding position of the human face can be obtained through the facial key point detection algorithm, and then the three-dimensional coordinates of the facial feature points in the vehicle coordinate system (X 2 , Y 2 , Z 2 );
  • the head reconstruction algorithm can also be used to directly reconstruct the user's head in real time, and then determine the three-dimensional coordinates (X 2 , Y 2 , Z 2 ) of the facial feature points in the vehicle coordinate system.
  • the display device in the embodiment of the present application is a holographic projection device, and the image displayed by it is a 3D image.
  • the direction of the displayed holographic image of the display device can be adjusted according to the first position information, so that the side of the holographic image containing the content that is intended to be conveyed to the user faces the user, for example, when the holographic image is a 3D virtual assistant
  • the virtual assistant's face, or body and face can be adjusted to face the user, as shown in (a) in Figure 6.
  • the virtual assistant's holographic image can be adjusted to face the co-pilot user.
  • the facial orientation of the digital person displayed by the display device can be adjusted according to the first position information.
  • the preset direction of the holographic image displayed by the display device may be as shown in (b) of Figure 6 .
  • the preset direction may be the virtual assistant.
  • the direction of the face; when the holographic image is another image, the default direction can be the direction of the side that contains the content you want to convey to the user.
  • adjusting the display direction of the display device according to the first position information may include: adjusting the pose of the holographic image so that the side containing the content intended to be conveyed to the user turns to the first position of the feature point of the user's face, that is, , point the preset direction shown in (b) in Figure 6 to the first position of the facial feature point, specifically as shown in (c) in Figure 6 .
  • the pose of the holographic image can be adjusted by adjusting the angle of the holographic projection device.
  • the holographic projection device includes a projection device and an imaging device, wherein the projection device projects light to the imaging device so that the imaging device can display a three-dimensional stereoscopic image.
  • the rotation angle of the projection device in a plane perpendicular to the vehicle's longitudinal symmetry plane can be adjusted according to the first position information, so that the holographic image contains the desired The side of the content conveyed to the user faces the user.
  • the holographic projection device rotates clockwise by ⁇ degree (degree, °) in a plane perpendicular to the longitudinal symmetry plane of the vehicle.
  • the holographic image will be rotated clockwise by ⁇ degrees in a plane perpendicular to the longitudinal symmetry plane of the vehicle, where ⁇ and ⁇ may be equal or unequal, and this is not limited in the embodiment of the present application.
  • the preset direction shown in (b) in Figure 6 indicates the initial pose of the holographic image.
  • the preset direction indicates the attitude angle of the holographic image as 0 degrees
  • the coordinates of the holographic image in the vehicle coordinate system are is (X3, Y3, Z3)
  • the coordinates of the first position shown in (c) in Figure 6 in the vehicle coordinate system are (X4, Y4, Z4), then according to the first position information and the initial position of the holographic image
  • the first rotation angle ⁇ 1 can be determined as arctan ((X4-X3)/(Y4-Y3)), and the first rotation direction is clockwise rotation in a plane perpendicular to the vehicle's longitudinal symmetry plane.
  • the rotation angle ⁇ 2 of the holographic projection device is determined, and then the holographic projection device is controlled to rotate clockwise in a plane perpendicular to the vehicle's longitudinal symmetry plane. ⁇ 2 , so that the side of the hologram containing the content intended to be conveyed to the user faces the user.
  • the above-mentioned projection device can realize digital high-definition multi-channel hard disk synchronous playback;
  • the above-mentioned imaging device can include a spectroscope.
  • the above-mentioned projection device and imaging device may also include other devices, which are not specifically limited in the embodiments of the present application.
  • the pose of the three-dimensional image can also be adjusted according to the first position information.
  • the orientation of the virtual assistant's head and/or body can be adjusted so that the head of the virtual assistant faces the first position.
  • the height of the image displayed by the holographic projection device can also be adjusted according to the first position information.
  • the height of the image displayed by the holographic projection device is adjusted according to the height information in the first position information, thereby controlling the holographic projection device to display the holographic image at a corresponding height.
  • the rotation angle of the projection device in a plane parallel to the vehicle's longitudinal symmetry plane, and thus the height of the holographic image can be controlled based on the height information in the first position.
  • S405 Obtain the user's second facial image, and determine the second position information of the facial feature points based on the second facial image.
  • the second position information is used to indicate the second position of the facial feature point in the vehicle coordinate system.
  • the user's head position changes.
  • the user's second facial image can be obtained, and the third facial feature point of the user can be determined based on the second facial image.
  • Location information Specifically, for the method of determining the second position information of the facial feature points, reference may be made to the description in S403, which will not be described again here.
  • S406 Determine whether the distance difference between the first position and the second position is greater than or equal to the first threshold.
  • S407 when it is determined that the distance difference between the first position and the second position is greater than or equal to the first threshold, S407 is executed; otherwise, S408 is executed.
  • the above-mentioned first threshold may be 15 centimeters (cm), or 20 cm, or may be other distances, which are not specifically limited in the embodiments of the present application.
  • both the first position and the second position are the positions of the facial feature points of the first user (such as the main driver) in the vehicle coordinate system. Then, adjusting the display direction of the display device according to the second position information may be as shown in (d) of Figure 6 .
  • the first position is the position of the face feature point of the first user (such as the user at the main driver's seat) in the vehicle coordinate system
  • the second position is the position of the second user (such as the user at the passenger seat) The position of the facial feature points in the vehicle coordinate system.
  • adjusting the display direction of the display device according to the second position information can be as shown in (e) in Figure 6 .
  • the determination of the second location may be actively triggered by the second user.
  • the second user triggers through a voice command, such as "Hologram, please face me"; or the second user triggers through a related button, such as the second user clicks a button that "turns on the hologram".
  • the angle and direction in which the display device needs to be rotated can be determined based on the first position information and the second position information, and then the display device is controlled to rotate so that the display direction of the display device faces the second position.
  • the display device rotates in a plane perpendicular to the longitudinal plane of symmetry of the vehicle.
  • the steps or operations of the method of controlling a display device shown in FIG. 4 are only exemplary. Embodiments of the present application may also perform other operations or modifications of each operation in FIG. 4 . Additionally, the various steps in FIG. 4 may be performed in a different order than presented in FIG. 4 , and not all operations in FIG. 4 may be performed. For example, S405 to S407 may not be executed. For example, S402 can also be skipped and S403 can be executed directly.
  • the embodiments of the present application provide a method for adjusting a display device, which can adjust the display direction of the display device according to the three-dimensional coordinates of the characteristic points of the user's head in the vehicle coordinate system, thereby accurately adjusting the image displayed by the display device.
  • the orientation provides users with better interactive effects and improves users’ interactive experience.
  • Figure 7 shows a method for adjusting a display device provided by an embodiment of the present application.
  • the method 700 can be applied to the scenario shown in Figure 1 or to the vehicle 100 shown in Figure 2. This method can also be applied to the scenario shown in Figure 1. It is executed by the system shown in Figure 3.
  • the steps or operations of the method of controlling a display device shown in FIG. 7 are only exemplary. Embodiments of the present application may also perform other operations or modifications of each operation in FIG. 4 .
  • the method 700 includes:
  • S701 Obtain the audio data inside the vehicle and determine the sound source location of the audio data.
  • the audio information may be audio information obtained by excluding various invalid audio information from the collected audio information inside the vehicle.
  • the invalid audio information may be audio information with a volume that is too low.
  • the sound source position may be the position of the sound source corresponding to the audio information.
  • the sound source position may be the relative position to the light display device based on sound source positioning, or it may be specific position coordinates. This is not specified in the embodiment of this application. limit.
  • the sound source location can be determined based on the audio information collected by multiple sound sensors based on the time difference of arrival (TDOA) principle.
  • TDOA time difference of arrival
  • sound sensors A and B respectively detect that audio is emitted from sound source S.
  • the time when the sound source signal of sound source S reaches sound sensor A is t1
  • the time when it reaches sound sensor B is t2.
  • the time difference dt
  • the audio information can be a voice containing a specific wake-up word, such as "Turn on the display device"; or, “Virtual assistant, please face me”; or it can also be other voice information.
  • S702 Determine the user's position based on the sound source position.
  • two sound source locations are determined based on the acquired audio data, for example, the sound source locations are determined to be the main driver's and co-pilot's locations, and the display device can be controlled to "copy" the image. ", the two images obtained after controlling the copy are respectively facing the user at the main driver's seat and the user at the co-pilot's seat, as shown in (a) in Figure 8. For example, if the image is a 3D image of a virtual assistant, then the two copied images are controlled by the same command.
  • two sound source locations are determined based on the acquired audio data. For example, if the sound source locations are determined to be the main driver's and co-pilot's locations, the display device can be controlled to display two images. , and control the above two images to face the main driver's user and the co-pilot's user respectively.
  • the above-mentioned preset time length may be 3 seconds, or 5 seconds, or may be other preset time lengths.
  • the display device is set at the center armrest of the front row.
  • the display device can be controlled to project two images. The two images are respectively facing the user in the left seat in the back row and the user in the right seat in the back row.
  • the display device when the number of users is greater than 2, the display device can also be controlled to project multiple images, and the multiple images can be adjusted to face multiple users respectively.
  • the display device can be adjusted so that the image projected by the display device moves to the sound source position. For example, if the sound source position is at the passenger's side, the display device can be adjusted so that the position of the image moves to the passenger's side, as shown in (b) of Figure 8 .
  • the holographic projection device includes a projection device and an imaging device, wherein the projection device projects light to the imaging device so that the imaging device can display a three-dimensional stereoscopic image.
  • the position of the imaging device is adjusted to the passenger's position, and the projection angle of the projection device is adaptively adjusted, thereby forming a holographic image at the passenger's position.
  • the embodiments of this application provide a method for adjusting a display device.
  • a user issues a voice command
  • it can be determined based on the sound source position whether the user issuing the command is the driver, co-driver or rear passenger, and then adjusts the corresponding
  • the display device enables the 3D image projected by the display device to face the user, thereby improving the interactivity between the user and the image projected by the display device.
  • the display device can also copy images, so that each image faces one of multiple users, further improving the user's interactive experience and driving fun.
  • Figure 9 shows a method for adjusting a display device provided by an embodiment of the present application.
  • the method 900 can be applied to the scenario shown in Figure 1 or to the vehicle 100 shown in Figure 2. This method can also be applied to the scenario shown in Figure 1. It is executed by the system shown in Figure 3.
  • the steps or operations of the method of controlling a display device shown in FIG. 9 are only exemplary. Embodiments of the present application may also perform other operations or modifications of each operation in FIG. 4 .
  • the method 900 includes:
  • S902 Determine the first spatial coordinates of the first user's facial feature points based on the first facial image.
  • the above-mentioned first spatial coordinates may be the three-dimensional coordinates of the first user's facial feature points in the vehicle coordinate system.
  • the facial feature points of the first user may be the points in the above embodiment.
  • the method of determining the first spatial coordinate reference may be made to the description in the above embodiment, which will not be described again here.
  • S903 Adjust the orientation of the first side of the image displayed by the display device according to the first spatial coordinate.
  • the first side of the image contains information to be conveyed to the first user.
  • the image may be a holographic image, or it may be a two-dimensional image with a three-dimensional effect and depth of field, which is not specifically limited in the embodiments of the present application.
  • the method of adjusting a display device allows users to experience interaction with the display device, which helps improve the user experience at the interactive level.
  • Figure 10 shows a schematic block diagram of a device 2000 for adjusting a display device provided by an embodiment of the present application.
  • the device 2000 includes an acquisition unit 2010 and a processing unit 2020.
  • the acquisition unit 2010 can implement corresponding communication functions, and the processing unit 2020 is used for data processing.
  • the device 2000 may also include a storage unit, which may be used to store instructions and/or data, and the processing unit 2020 may read the instructions and/or data in the storage unit, so that the device implements the foregoing method embodiments. .
  • the apparatus 2000 may include units for performing the methods in FIG. 4, FIG. 7, and FIG. 9. Moreover, each unit in the device 2000 and the above-mentioned other operations and/or functions are respectively intended to implement the corresponding processes of the method embodiments in FIG. 4, FIG. 7, and FIG. 9.
  • the acquisition unit 2010 can be used to execute S901 in the method 900
  • the processing unit 2020 can be used to execute S902 and S903 in the method 900.
  • the device 2000 includes: an acquisition unit 2010, used to acquire a first facial image of a first user; a processing unit 2020, used to determine first spatial coordinates of facial feature points of the first user based on the first facial image. ; Adjust the orientation of the first side of the image displayed by the display device according to the first spatial coordinate, and the first side of the image contains information to be conveyed to the first user.
  • the acquisition unit 2010 is also configured to: acquire the second facial image of the second user after adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinate;
  • the processing unit 2020 is also configured to: : Determine the second spatial coordinates of the facial feature points of the second user based on the second facial image; adjust the orientation of the first side of the image displayed by the display device based on the second spatial coordinates.
  • the acquisition unit 2010 is also configured to: acquire the third facial image of the first user after adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinate;
  • the processing unit 2020 is also configured to: In: determining the third spatial coordinates of the first user's facial feature points based on the third facial image; when the distance between the first spatial coordinates and the third spatial coordinates is greater than or equal to a preset threshold, based on the The third spatial coordinate adjusts the orientation of the first side of the image displayed by the display device.
  • the acquisition unit 2010 is further configured to: before acquiring the first facial image of the first user, acquire audio information, where the audio information includes a voice instruction of the first user, and the voice instruction is used to indicate turning on and /or adjust the display device; the processing unit 2020 is also used to: determine the sound source position of the audio information; determine the first user's position according to the sound source position.
  • the audio information also includes a second user's voice instruction
  • the processing unit 2020 is further configured to: control the display device to display the first image and the second image; adjust the display device so that the first side of the first image toward the first user, and the first side of the second image faces the second user, wherein the image displayed by the display device includes the first image and/or the second image, and the first side of the first image Containing information to be conveyed to the first user, the first side of the second image contains information to be conveyed to the second user.
  • the processing unit 2020 is also configured to adjust the position of the image displayed by the display device according to the first spatial coordinate.
  • the acquisition unit 2010 is further configured to: before adjusting the orientation of the first side of the image displayed by the display device according to the first spatial coordinate, acquire the initial pose information of the display device, where the initial pose information is used Indicate the posture angle and position of the display device; the processing unit 2020 is also configured to: determine the rotation angle and rotation direction of the display device according to the first spatial coordinate and the initial posture information; according to the rotation angle and the rotation direction, Adjust the orientation of the first side of the image displayed by the display device.
  • the first spatial coordinates of the first user's facial feature points are used to indicate the three-dimensional position of the first user's facial feature points in the vehicle, and the vehicle includes the display device.
  • the display device includes a holographic projection device, and the image includes a three-dimensional 3D image.
  • each unit in the above device is only a division of logical functions.
  • the units may be fully or partially integrated into a physical entity, or may be physically separated.
  • the unit in the device can be implemented in the form of a processor calling software; for example, the device includes a processor, the processor is connected to a memory, instructions are stored in the memory, and the processor calls the instructions stored in the memory to implement any of the above methods.
  • the processor is, for example, a general-purpose processor, such as a CPU or a microprocessor
  • the memory is a memory within the device or a memory outside the device.
  • the units in the device can be implemented in the form of hardware circuits, and some or all of the functions of the units can be implemented through the design of the hardware circuits, which can be understood as one or more processors; for example, in one implementation,
  • the hardware circuit is an ASIC, which realizes the functions of some or all of the above units through the design of the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented through PLD, taking FPGA as an example. It can include a large number of logic gate circuits, and the connection relationships between the logic gate circuits can be configured through configuration files to realize the functions of some or all of the above units. All units of the above device may be fully realized by the processor calling software, or may be fully realized by hardware circuits, or part of the units may be realized by the processor calling software, and the remaining part may be realized by hardware circuits.
  • the processor is a circuit with signal processing capabilities.
  • the processor may be a circuit with instruction reading and execution capabilities, such as a CPU, a microprocessor, a GPU, or DSP, etc.; in another implementation, the processor can realize certain functions through the logical relationship of the hardware circuit. The logical relationship of the hardware circuit is fixed or can be reconstructed.
  • the processor is a hardware circuit implemented by ASIC or PLD. For example, FPGA.
  • the process of the processor loading the configuration file and realizing the hardware circuit configuration can be understood as the process of the processor loading instructions to realize the functions of some or all of the above units.
  • it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as NPU, TPU, DPU, etc.
  • each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA , or a combination of at least two of these processor forms.
  • processors or processing circuits
  • each unit in the above device may be integrated together in whole or in part, or may be implemented independently. In one implementation, these units are integrated together and implemented as a system-on-a-chip (SOC).
  • SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device.
  • the at least one processor may be of different types, such as a CPU and an FPGA, or a CPU and an artificial intelligence processor. CPU and GPU etc.
  • each operation performed by the above-mentioned acquisition unit 2010 and processing unit 2020 can be performed by the same processor, or can also be performed by different processors, for example, by multiple processors.
  • one or more processors can be connected to one or more sensors in the sensing system 120 in Figure 2 to obtain the user's facial image from one or more sensors and process it; in another example, one or more The processor may also be connected to one or more display devices in the display device 130 to control the orientation of images displayed by the display devices.
  • the one or more processors described above may be processors provided in a vehicle machine, or may also be processors provided in other vehicle-mounted terminals.
  • the above-mentioned device 2000 may be a chip provided in a vehicle machine or other vehicle-mounted terminal.
  • the above-mentioned device 2000 may be the computing platform 150 as shown in FIG. 2 provided in the vehicle.
  • FIG. 11 is a schematic block diagram of a device for adjusting a display device according to an embodiment of the present application.
  • the apparatus 2100 for adjusting a display device shown in FIG. 11 may include: a processor 2110, a transceiver 2120, and a memory 2130.
  • the processor 2110, the transceiver 2120 and the memory 2130 are connected through an internal connection path.
  • the memory 2130 is used to store instructions.
  • the processor 2110 is used to execute the instructions stored in the memory 2130, and the transceiver 2120 receives/sends some parameters.
  • the memory 2130 can be coupled with the processor 2110 through an interface or integrated with the processor 2110 .
  • transceiver 2120 may include but is not limited to a transceiver device such as an input/output interface to realize communication between the device 2100 and other devices or communication networks.
  • the processor 2110 may use a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits to execute relevant programs to implement the method for adjusting a display device in the method embodiment of the present application.
  • the processor 2110 may also be an integrated circuit chip with signal processing capabilities.
  • each step of the method of adjusting a display device of the present application can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 2110 .
  • the above-mentioned processor 2110 can also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.
  • Each method, step and logical block diagram disclosed in the embodiment of this application can be implemented or executed.
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
  • the steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field.
  • the storage medium is located in the memory 2130.
  • the processor 2110 reads the information in the memory 2130 and executes the method for adjusting the display device according to the method embodiment of the present application in conjunction with its hardware.
  • the memory 2130 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).
  • ROM read-only memory
  • RAM random access memory
  • the transceiver 2120 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 2100 and other devices or communication networks. For example, the user's facial image and/or audio information may be obtained through the transceiver 2120.
  • An embodiment of the present application also provides a vehicle, which may include the above device 2000, or the above device 2100, and the above display device.
  • the display device can be a holographic projection device, and the displayed image can be a three-dimensional image; or the display device can also be a vehicle display screen, and the displayed image can be a digital human; or the display device can also be other display devices. , this application does not specifically limit this.
  • Embodiments of the present application also provide a computer program product.
  • the computer program product includes: computer program code.
  • the computer program code When the computer program code is run on a computer, it causes the computer to execute the above method.
  • Embodiments of the present application also provide a computer-readable storage medium.
  • the computer-readable medium stores program code.
  • the computer program code When the computer program code is run on a computer, it causes the computer to execute the above-mentioned steps in Figures 4, 7, and 9. Either way.
  • An embodiment of the present application also provides a chip, including: at least one processor and a memory.
  • the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to execute the above-mentioned Figures 4, 7, Either method in Figure 9.
  • At least one refers to one or more
  • plural refers to two or more.
  • “And/or” describes the association of associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural.
  • the character “/” generally indicates that the related objects are in an “or” relationship.
  • “At least one of the following” or similar expressions thereof refers to any combination of these items, including any combination of a single item (items) or a plurality of items (items).
  • At least one of a, b, or c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple .
  • the disclosed systems, devices and methods can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented.
  • the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit.
  • the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product.
  • the computer software product is stored in a storage medium and includes a number of instructions to enable a computer device (which can be a personal computer, a server, or network equipment, etc.) to perform all or part of the steps of the methods described in various embodiments of this application.
  • the aforementioned storage media include: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

本申请提供了一种调节显示设备的方法和装置，该方法包括：获取第一用户的第一面部图像；根据该第一面部图像确定该第一用户的人脸特征点的第一空间坐标；根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向，该影像的第一侧包含要传达给该第一用户的信息。本申请的调节显示设备的方法，能够根据用户的位置精准的调节显示设备显示的影像的朝向，进而让用户体会到与显示设备的交互，有助于在交互层面提高用户的使用体验。

Description

调节显示设备的方法和装置
本申请要求于2022年5月10日提交中国专利局、申请号为202210505723.1、申请名称为“调节显示设备的方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人机交互领域，更具体地，涉及一种调节显示设备的方法和装置。
背景技术
随着智能汽车的发展，越来越多的成像显示设备被集成到座舱中来，例如车载大屏、抬头显示设备(head up display，HUD)、车载全息影像等显示设备。但现有的显示设备显示范围均有限，当用户处于不同位置时，对于该用户而言，这些显示设备并不一定处于较佳的显示角度或位置，甚至出现成像不完全、显示不可见的情况。
因此，一种能够根据用户位置精准地调节显示设备的方法和装置亟待开发。
发明内容
本申请提供一种调节显示设备的方法和装置，能够根据用户的位置精准地调节显示设备显示的影像的朝向，有助于在人机交互层面提高用户的使用体验。
第一方面，提供了一种调节显示设备的方法，该方法可以由车辆执行；或者，也可以由车辆的车载终端如车机等执行；或者，还可以由用于车辆的芯片或电路执行，本申请对此不作限定。为了便于描述，下面以车辆执行为例进行说明。
该方法可以包括：获取第一用户的第一面部图像；根据该第一面部图像确定该第一用户的人脸特征点的第一空间坐标；根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向，该影像的第一侧包含要传达给该第一用户的信息。
在上述技术方案中，根据用户的位置精准的调节显示设备显示的影像的朝向，能够在交互层面提高用户的使用体验。
在一些可能的实现方式中，上述第一空间坐标为用户的人脸特征点在整车坐标系中的三维坐标。
示例性地，人脸特征点可以位于第一用户的眉心、眼睛、眉毛、鼻子中的任意一处或几处；或者，人脸特征点还可以为第一用户面部其他位置的点，本申请实施例对此不作具体限定。
在一些可能的实现方式中，上述影像可以为三维影像，如全息投影。应理解，全息投影的不同角度所包含的信息不同。或者，上述影像也可以为与第一用户进行交互的、具有三维特征的二维图像，例如数字人。应理解，数字人的面部朝向不同时，能够实现不同的交互效果，例如，数字人面部朝向第一用户时，能够使第一用户感受到与数字人面对面交 流的效果。
一示例，该影像为虚拟助手的三维影像，则该影像的第一侧可以为包含虚拟助手的脸部正面的一侧。又一示例，该影像为自车的三维影像，要向第一用户展示车辆前围罩的样式，则该影像的第一侧可以为包含车辆前围罩的一侧。
示例性地，调节影像的第一侧的朝向可以包括调节影像的姿态，例如，影像为虚拟助手的三维影像时，可以调节影像的头部姿态，以使其脸部正面朝向第一用户。
结合第一方面，在第一方面的某些实现方式中，该方法还包括：该根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，获取第二用户的第二面部图像；根据该第二面部图像确定该第二用户的人脸特征点的第二空间坐标；根据该第二空间坐标调节该显示设备显示的该影像的第一侧的朝向。
在一些可能的实现方式中，第二空间坐标的确定可以是由第二用户主动触发的。例如，第二用户通过语音指令触发，例如，第二用户发出语音指令“全息影像，请面向我”；或者，第二用户通过相关按钮触发，例如第二用户点击“开启全息影像”的按钮。进一步地，车辆控制摄像装置获取第二用户的第二面部图像，进而确定第二用户的人脸特征点的第二空间坐标。
在一些可能的实现方式中，该根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之后的预设时长外，获取第二用户的第二面部图像。示例性地，上述预设时长可以为3秒，或者5秒，或者也可以为其他预设时长，本申请实施例对此不作具体限定。
在上述技术方案中，在存在两个及以上多用户时，可以根据多个用户中每个用户的人脸特征点的三维坐标，进行显示设备显示影像的朝向的调节，提高多用户场景下的交互体验。
结合第一方面，在第一方面的某些实现方式中，该方法还包括：该根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，获取该第一用户的第三面部图像；根据该第三面部图像确定该第一用户的人脸特征点的第三空间坐标；在该第一空间坐标与该第三空间坐标之间的距离大于或等于预设阈值时，根据该第三空间坐标调节该显示设备显示的该影像的第一侧的朝向。
在一些可能的实现方式中，在第一用户与影像进行交互时，第一用户的头部位置发生了变化，此时可以获取第一用户的第二面部图像，并根据第二面部图像确定人脸特征点的第二坐标位置。
进一步地，在第一用户头部移动距离超过预设阈值时，根据第二空间坐标调节影像的第一侧的朝向。
示例性地，预设阈值可以为15厘米，或者20厘米，或者也可以为其他距离，本申请实施例对此不作具体限定。
在上述技术方案中，当用户头部位置移动的距离超过预设距离时，根据其移动后的位置调节影像的朝向，能够避免频繁调节影像的朝向，有助于提高用户的驾乘体验。
结合第一方面，在第一方面的某些实现方式中，该方法还包括：该获取第一用户的第一面部图像之前，获取音频信息，该音频信息包括该第一用户的语音指令，该语音指令用于指示开启和/或调节该显示设备；确定该音频信息的声源位置；根据该声源位置确定该第一用户的位置。
在上述技术方案中，在多用户场景下，当一个用户发出语音指令时，可以通过声源位置判定发出指令的用户是主驾、副驾还是后排乘客，进而调节相应的显示设备，以使显示设备显示的影像的包含想要传达给用户信息的一侧朝向该用户，提高用户与显示设备显示的影像之间的交互性。
结合第一方面，在第一方面的某些实现方式中，该音频信息还包括第二用户的语音指令，该方法还包括：控制该显示设备显示第一影像和第二影像；调节该显示设备使该第一影像的第一侧朝向该第一用户，且该第二影像的第一侧朝向该第二用户，其中，该显示设备显示的该影像包括该第一影像和/或第二影像，该第一影像的第一侧包含要传达给该第一用户的信息，该第二影像的第一侧包含要传达给该第二用户的信息。
在上述技术方案中，在多用户场景下，显示设备还可以进行影像复制，进而使每个影像分别面对多个用户中的一个用户，进一步提升用户的交互体验和驾乘趣味性。
结合第一方面，在第一方面的某些实现方式中，该方法还包括：根据该第一空间坐标调节该显示设备显示的该影像的位置。
在一些可能的实现方式中，可以根据第一空间坐标调节影像在整车坐标系中y轴方向上的位置，例如，在确定用户的位置后，可以调节显示设备，以使显示设备投射的影像移动与用户的位置相对应的位置处。示例性地，用户位置为副驾驶处，则可调节显示设备，以使影像的位置移动到副驾驶处。
在一些可能的实现方式中，可以根据第一空间坐标调节影像在整车坐标系中z轴方向上的位置，例如，在确定用户的位置后，可以调节显示设备，以使显示设备投射的影像的高度移动到适合用户观看的高度处。
应理解，整车坐标系中的y轴方向，为垂直于车辆纵向对称平面的方向；整车坐标系中的z轴方向，为平行于车辆纵向对称平面的方向。
在上述技术方案中，调节显示设备，以控制影像移动至用户所处位置，或者，控制影像移动至合适的高度，使得用户不必转头跟显示设备显示的影像进行交互，有助于提高用户的驾乘体验。
结合第一方面，在第一方面的某些实现方式中，该方法还包括：该根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之前，获取该显示设备的初始位姿信息，该初始位姿信息用于指示该显示设备的姿态角和位置；该根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向，包括：根据该第一空间坐标和该初始位姿信息确定该显示设备的旋转角度和旋转方向；根据该旋转角度和该旋转方向调节该显示设备显示的该影像的第一侧的朝向。
在一些可能的实现方式中，在根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之前，获取该影像的初始位姿信息，该影像的初始位姿信息用于指示该影像的姿态角和位置；根据该第一空间坐标和该影像的初始位姿信息确定该影像的旋转角度和旋转方向。进一步地，根据显示设备的角度变换与影像的角度变换之间的映射关系，确定显示设备的旋转角度和旋转方向，进而调节显示设备。示例性地，上述影像为虚拟助手，则影像的姿态角可以为虚拟助手的头部姿态角。
结合第一方面，在第一方面的某些实现方式中，该第一用户的人脸特征点的第一空间坐标用于指示该第一用户的人脸特征点在车辆中的三维位置，该车辆包括该显示设备。
在上述技术方案中，能够根据用户的人脸特征点在整车坐标系中的三维坐标对显示设备进行调节，从而能够精确调整显示设备显示的影像的朝向，为用户提供更好的交互效果，提示用户的交互体验。
结合第一方面，在第一方面的某些实现方式中，该显示设备包括全息投影设备，该影像包括三维3D影像。
第二方面，提供了一种调节显示设备的装置，该装置包括：获取单元，用于获取第一用户的第一面部图像；处理单元，用于根据该第一面部图像确定该第一用户的人脸特征点的第一空间坐标；根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向。
结合第二方面，在第二方面的某些实现方式中，该获取单元还用于：在根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，获取第二用户的第二面部图像；该处理单元还用于：根据该第二面部图像确定该第二用户的人脸特征点的第二空间坐标；根据该第二空间坐标调节该显示设备显示的该影像的第一侧的朝向。
结合第二方面，在第二方面的某些实现方式中，该获取单元还用于：在根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，获取该第一用户的第三面部图像；该处理单元还用于：根据该第三面部图像确定该第一用户的人脸特征点的第三空间坐标；在该第一空间坐标与该第三空间坐标之间的距离大于或等于预设阈值时，根据该第三空间坐标调节该显示设备显示的该影像的第一侧的朝向。
结合第二方面，在第二方面的某些实现方式中，该获取单元还用于：在该获取第一用户的第一面部图像之前，获取音频信息，该音频信息包括该第一用户的语音指令，该语音指令用于指示开启和/或调节该显示设备；该处理单元还用于：确定该音频信息的声源位置；根据该声源位置确定该第一用户的位置。
结合第二方面，在第二方面的某些实现方式中，该音频信息还包括第二用户的语音指令，该处理单元还用于：控制该显示设备显示第一影像和第二影像；调节该显示设备使该第一影像的第一侧朝向该第一用户，且该第二影像的第一侧朝向该第二用户，其中，该显示设备显示的该影像包括该第一影像和/或第二影像，该第一影像的第一侧包含要传达给该第一用户的信息，该第二影像的第一侧包含要传达给该第二用户的信息。
结合第二方面，在第二方面的某些实现方式中，该处理单元还用于：根据该第一空间坐标调节该显示设备显示的该影像的位置。
结合第二方面，在第二方面的某些实现方式中，该获取单元还用于：在根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之前，获取该显示设备的初始位姿信息，该初始位姿信息用于指示该显示设备的姿态角和位置；该处理单元还用于：根据该第一空间坐标和该初始位姿信息确定该显示设备的旋转角度和旋转方向；根据该旋转角度和该旋转方向，调节该显示设备显示的该影像的第一侧的朝向。
结合第二方面，在第二方面的某些实现方式中，该第一用户的人脸特征点的第一空间坐标用于指示该第一用户的人脸特征点在车辆中的三维位置，该车辆包括该显示设备。
结合第二方面，在第二方面的某些实现方式中，该显示设备包括全息投影设备，该影像包括三维3D影像。
第三方面，提供了一种调节显示设备的装置，该装置包括：存储器，用于存储程序；处理器，用于执行存储器存储的程序，当存储器存储的程序被执行时，处理器用于执行上 述第一方面中任一种可能实现方式中的方法。
第四方面，提供了一种车辆，该车辆包括上述第二方面中任一种可能实现方式中的装置或者上述第三方面中任一种实现方式中的装置，以及上述显示设备。其中，该显示设备可以为全息投影设备，显示的影像可以为三维影像；或者，该显示设备也可以为车载显示屏，显示的影像可以为数字人；或者，该显示设备也可以为其他显示设备，本申请对此不作具体限定。
第五方面，提供了一种计算机程序产品，上述计算机程序产品包括：计算机程序代码，当上述计算机程序代码在计算机上运行时，使得计算机执行上述第一方面中任一种可能实现方式中的方法。
需要说明的是，上述计算机程序代码可以全部或部分存储在第一存储介质上，其中第一存储介质可以与处理器封装在一起的，也可以与处理器单独封装，本申请实施例对此不作具体限定。
第六方面，提供了一种计算机可读介质，上述计算机可读介质存储由程序代码，当上述计算机程序代码在计算机上运行时，使得计算机执行上述第一方面中任一种可能实现方式中的方法。
第七方面，提供了一种芯片，该芯片包括处理器，用于调用存储器中存储的计算机程序或计算机指令，以使得该处理器执行上述第一方面中任一种可能实现方式中的方法。
结合第七方面，在一种可能的实现方式中，该处理器通过接口与存储器耦合。
结合第七方面，在一种可能的实现方式中，该芯片系统还包括存储器，该存储器中存储有计算机程序或计算机指令。
本申请实施例提供的一种调节显示设备的方法，能够根据用户的位置精准的调节显示设备显示的影像的朝向，进而在交互层面提高用户的使用体验。能够根据用户头部移动的位置进行显示设备调节，当用户头部位置移动的距离超过预设距离时，根据其移动后的位置调节影像的朝向，能够避免频繁调节影像的朝向，有助于提高用户的驾乘体验。在存在两个及以上多用户时，可以根据多个用户中每个用户的人脸特征点的三维坐标，进行显示设备显示影像的朝向的调节，提高多用户场景下的交互体验。还能够根据用户发出的语音指令确定用户的位置，进而根据用户的位置调节显示设备。在多用户场景下，当一个用户发出语音指令时，可以通过声源位置判定发出指令的用户是主驾、副驾还是后排乘客，进而调节相应的显示设备，以使显示设备显示的影像的包含想要传达给用户信息的一侧朝向该用户，提高用户与显示设备显示的影像之间的交互性。在多用户场景下，显示设备还可以进行影像复制，进而使每个影像分别面对多个用户中的一个用户，进一步提升用户的交互体验和驾乘趣味性。还能够通过调节显示设备，控制影像移动至用户所处位置，或者，控制影像移动至合适的高度，使得用户不必转头跟显示设备显示的影像进行交互，有助于提高用户的驾乘体验。
附图说明
图1是本申请实施例提供的一种调节显示设备的方法的应用场景示意图。
图2是本申请实施例提供的车辆100的示意性框图。
图3是本申请实施例提供的一种调节显示设备的系统的示意性框图。
图4是本申请实施例提供的一种调节显示设备的方法的示意性流程图。
图5是本申请实施例提供的图像坐标系与相机坐标系间的变换关系的示意图。
图6是本申请实施例提供的调节显示设备显示的影像的应用场景示意图。
图7是本申请实施例提供的一种调节显示设备的方法的又一示意性流程图。
图8是本申请实施例提供的调节显示设备显示的影像的又一应用场景示意图。
图9是本申请实施例提供的一种调节显示设备的方法的再一示意性流程图。
图10是本申请实施例提供的一种调节显示设备的装置的示意性框图。
图11是本申请实施例提供的一种调节显示设备的装置的示意性框图。
具体实施方式
下面将结合附图，对本申请实施例中的技术方案进行描述。
图1为本申请实施例提供的一种车辆座舱场景的示意图。智能座舱内部可以安装有一个或多个摄像头，用于捕捉舱内或舱外的图像，例如，驾驶员监测系统(driver monitor system，DMS)的摄像头，座舱监测系统(cabin monitor system，CMS)的摄像头，以及行车记录仪(dashcam)的摄像头。图1中的(a)中以设置在A柱的摄像头为例。其中，用于捕捉舱内和舱外的摄像头可以是同一个摄像头，也可以是不同摄像头。此外，座舱内还设置有车载显示屏，图1中的(a)中以设置于中控区域的显示屏为例。车辆可以通过车载显示屏、HUD(图1中的(a)中未画出)以及车载全息投影设备(图1中的(a)中未画出)中的至少一个向用户显示信息。示例性地，通过车载全息投影设备可以为用户显示虚拟助手的三维影像，具体如图1中的(a)所示。在一些可能的实现方式中，车载全息投影设备还可以在前排座椅头枕后部进行成像，具体如图1中的(b)所示。应理解，图1中的(a)所示的三维影像仅为示例性说明，三维影像的具体内容还可以为其他三维物体，例如还可以为自车的三维影像。应理解，本申请实施例中座舱内采集图像信息的摄像头的位置并不作具体限定。摄像头可以位于图1中的(a)所示的A柱上，也可以位于B柱上，也可以位于方向盘下方，还可以位于后视镜附近等位置。
图2是本申请实施例提供的车辆100的一个功能框图示意。车辆100可以包括感知系统120、显示装置130和计算平台150。感知系统120可以包括感测关于车辆100周边的环境的信息以及车辆座舱内部环境的若干种传感器。例如，感知系统120可以包括定位系统，定位系统可以是全球定位系统(global positioning system，GPS)，也可以是北斗系统或者其他定位系统、惯性测量单元(inertial measurement unit，IMU)、激光雷达、毫米波雷达、超声雷达、视觉传感器、声音传感器以及摄像装置中的一种或者多种。上述摄像装置可以包括红绿蓝/红外线(red,green and blue/infrared，RGB/IR)摄像头，也可以包括深度相机，如飞行时间(time of flight，TOF)相机、双目相机、结构光相机等。
车辆100的部分或所有功能可以由计算平台150控制。计算平台150可包括处理器151至15n(n为正整数)，处理器是一种具有信号的处理能力的电路，在一种实现中，处理器可以是具有指令读取与运行能力的电路，例如中央处理单元(central processing unit，CPU)、微处理器、图形处理器(graphics processing unit，GPU)(可以理解为一种微处理器)、或数字信号处理器(digital signal processor，DSP)等；在另一种实现中，处理器 可以通过硬件电路的逻辑关系实现一定功能，该硬件电路的逻辑关系是固定的或可以重构的，例如处理器为专用集成电路(application-specific integrated circuit，ASIC)或可编程逻辑器件(programmable logic device，PLD)实现的硬件电路，例如现场可编辑逻辑门阵列(filed programmable gate array，FPGA)。在可重构的硬件电路中，处理器加载配置文档，实现硬件电路配置的过程，可以理解为处理器加载指令，以实现以上部分或全部单元的功能的过程。此外，还可以是针对人工智能设计的硬件电路，其可以理解为一种ASIC，例如神经网络处理单元(neural network processing unit，NPU)、张量处理单元(tensor processing unit，TPU)、深度学习处理单元(deep learning processing unit，DPU)等。此外，计算平台150还可以包括存储器，存储器用于存储指令，处理器151至15n中的部分或全部处理器可以调用存储器中的指令，执行指令，以实现相应的功能。
座舱内的显示装置130主要分为三类，第一类是车载显示屏；第二类是投影显示屏，例如HUD；第三类为全息投影设备。车载显示屏是一种物理显示屏，是车载信息娱乐系统的重要组成部分，座舱内可以设置有多块显示屏，如数字仪表显示屏，中控屏，副驾驶位上的乘客(也称为前排乘客)面前的显示屏，左侧后排乘客面前的显示屏以及右侧后排乘客面前的显示屏，甚至是车窗也可以作为显示屏进行显示。抬头显示，也称平视显示系统，包括组合型抬头显示(combiner-HUD，C-HUD)系统、风挡型抬头显示(windshield-HUD，W-HUD)系统、增强现实型抬头显示系统(augmented reality HUD，AR-HUD)。全景影像系统是一种能够显示物体三维(three dimension，3D)影像的显示装置。在一些可能的实现方式中，全息投影设备可以基于光的干涉和衍射原理记录并再现物体的3D影像，或者，还可以通过边缘消隐、珮珀尔幻像(Pepper’s ghost)等方法实现3D效果。在本申请实施例中，可以通过全息投影设备显示3D虚拟助手的影像，或者自车的3D影像，或者，还可以显示其他3D影像，本申请实施例对此不作具体限定。在一些可能的实现方式中，显示装置130还可以显示数字人(digitalhuman)，应理解，数字人可以指具有人类相貌、性格等人物特征，具有语言、面部表情和肢体动作表达能力，能够与人类交流互动的虚拟图像，该图像可以通过车载显示屏等进行显示。
应理解，HUD所显示的图像为投射在HUD屏幕上的二维图像；车载显示屏显示的数字人也包含于具有立体效果和景深的二维图像中；而全息投影设备所显示的影像为三维立体的影像，即从不同角度观察全息影像，可以看到物体的不同侧面，有视差效果和景深感。
在本申请实施例中，处理器可以获取感知系统120采集的用户图像信息，并结合车辆100的三维坐标系确定用户眼睛位置，以及用户的视线关注点，在确定用户的视线关注点在显示设备和/或显示设备显示的影像上时，根据用户在车辆100中的实际位置进行显示设备调节。在一些可能的实现方式中，处理器还可以根据感知系统120中声音传感器采集的声音信息确定用户所在位置，进而根据用户所在位置调节全息投影设备所投出影像的角度。在一些可能的实现方式中，上述用户图像信息和声音信息还可以以数据的形式存储于计算平台150中的存储器中。一些可能的实现方式中，处理器可以对上述用户图像信息和声音信息进行处理，获得参数化指标，进而根据参数化指标调节显示设备。应理解，上述操作可以由同一个处理器执行，或者，也可以由一个或多个处理器执行，本申请实施例对此不作具体限定。
以下结合图3详细描述调节显示设备的系统的工作流程。如图3所示为本申请实施 例提供的一种调节显示设备的系统架构图，该系统200中包括感知模块、标定模块、显示设备调整模块和显示模块。其中，感知模块可以包括图2中所示的感知系统120中的一种或多种摄像装置，以及一种或多种传感器；显示模块可以包括图2中所示的显示设备130中的一个或多个显示设备；标定模块和显示设备调整模块可以包括图2中所示的计算平台150中的一个或多个处理器。具体地，感知模块可以检测用户面部信息和/或声音信息，显示设备调整模块可以根据从感知模块获取的人脸图像信息和从标定模块获取的整车坐标系下的外参，确定调整参数，进而结合整车坐标系下的外参以及人脸图像信息确定用户面部任一位置在车辆内的实际三维坐标。进一步地，显示设备调整模块可以根据用户面部的某一个或某几个特征点的三维坐标对显示模块中的显示设备进行调节。示例性地，上述用户面部的某一特征点可以为眼睛，或者也可以为两眼中间的一点，或者也可以为其他点，本申请实施例对此不作具体限定。
应理解，上述模块及装置仅为一个示例，实际应用中，上述模块和装置有可能根据实际需要添加或删除。一示例，图3中显示模块和显示设备调整模块也可以合并为一个模块，即二者的功能由一个模块实现。
如上所述，当前技术背景下的显示设备显示范围均有限，当用户处于不同位置时，对于该用户而言，这些显示设备并不一定处于较佳的显示角度或位置，甚至出现成像不完全、显示不可见的情况。特别是对于全息投影设备而言，其投射出的三维影像方向固定，不同位置用户看到的影像不同，可能会导致想要获取影像所传递的信息的用户无法及时、全面地获取相关信息。鉴于此，本申请实施例提供一种调节显示设备的方法和装置，能够根据用户位置对全息投影设备进行调节，以使其投射出的三维影像能够根据用户所在位置进行精准调节，增强与用户之间的交互性，进而提高用户的驾乘体验。
图4示出了本申请实施例提供的一种调节显示设备的方法，该方法400可以应用于图1所示的场景中，也可以应用于图2所示的车辆100中，该方法也可以由图3所示的系统执行。该方法400包括：
S401，获取用户的第一面部图像。
示例性地，在显示设备开启后，可以通过设置在车辆内部的摄像头获取用户的第一面部图像，其中，第一面部图像中包括用户的眼部图像。例如，为了便于采集驾驶员的面部图像，摄像头可以设置在A柱上，以使摄像头可以捕捉到用户的双眼。应理解，用户可以为车辆座舱内的任一用户，例如，可以为驾驶员，也可以为副驾驶处用户，或者也可以为座舱后排用户。
S402，判断用户的视线关注点是否为显示影像。
具体地，在确定用户的视线关注点为显示影像时，执行S403；否则，执行S405。
在一些可能的实现方式中，显示设备为全息投影设备，其投射出的影像为虚拟3D影像，则判断用户的视线关注点是否在该虚拟3D影像上。
示例性地，可以通过视线估计(gazeestimation)技术或视线追踪技术等确定用户的视线关注点。
S403，根据第一面部图像确定人脸特征点的第一位置信息。
示例性地，第一位置信息用于指示人脸特征点在整车坐标系下的第一位置。
在一些可能的实现方式中，可以根据第一面部图像确定一个人脸特征点，或者也可以 根据第一面部图像确定两个及以上人脸特征点，本申请实施例对此不作具体限定。
示例性地，人脸特征点可以位于眉心、眼睛、眉毛、鼻子中的任意一处或几处；或者，人脸特征点还可以为用户面部其他位置的点，本申请实施例对此不作具体限定。
进一步地，根据人脸特征点在第一面部图像上的图像坐标确定该点在车辆座舱内部的三维坐标，即第一位置信息。
在具体实现过程中，也可以根据第一面部图像确定人脸特征区域，进而根据人脸特征区域确定第一位置信息。或者，也可以通过第一面部图像确定的其他特征点或区域确定第一位置信息，本申请对此不作具体限定。
根据图像坐标确定第一位置信息的详细方法可以参考下面的描述：
图像坐标系是在摄像机拍摄的图像上建立的坐标系。该图像坐标系可以以摄像机拍摄的图像的中心为原点，上述图像坐标系是二维坐标系。相机坐标系以相机的光心为坐标系原点，Xc，Yc轴平行于图像坐标系的x，y轴，相机的光轴为Zc轴，坐标系满足右手法则。相机的光心可理解为相机透镜的几何中心，上述相机坐标系为三维坐标系。图像坐标系与相机坐标系间的变换关系可以如图5所示。图5中，二维坐标系o-xy为摄像机拍摄的图像q的图像坐标系。其中，x轴和y轴的交点为o，即o为图像坐标系的原点，图像平面q上任一点的坐标可以用图像坐标(x，y)表示。三维坐标系O-XYZ为相机坐标系，相机坐标系中的任一平面可以称为空间平面。其中，空间平面Q为空间坐标系中的任一平面(即空间平面Q为O-XYZ中的任一平面)，空间平面Q上任一点的坐标可以用三维坐标(X，Y，Z)表示。其中，图像平面q与空间平面Q有如下变换关系：q＝sHQ。其中，q表示图像平面中的点的图像坐标，Q表示空间平面中的点的三维坐标，s为尺度比例因子，单应矩阵H＝MW。
其中，
Figure PCTCN2022137563-appb-000001
M是摄像机的内参矩阵，f x表示摄像机中透镜物理焦距长度与成像的每个单元尺寸在x方向的乘积。f y表示摄像机中透镜物理焦距长度与成像的每个单元尺寸在y方向的乘积。c x表示投影坐标中心相对光轴在x方向的偏移，c y表示投影坐标中心相对光轴在y方向的偏移。成像的每个单元尺寸可以是摄像机拍摄的图像中的一个像素。
其中，W＝[R，t]，W是摄像机的外参矩阵。R为3*3的正交单位矩阵，也称为旋转矩阵，t为三维的平移向量。若图像平面q与空间平面Q有上述变换关系，并且已知q上某一点的图像坐标，尺度比例因子以及单应矩阵H，可以根据上述图像平面与空间平面的变换关系计算该点在Q上的空间坐标。
例如，图5中，平面q上某一点的图像坐标为(x 1，y 1)，若摄像机的内参矩阵为M，摄像机的外参矩阵为W，尺度比例因子为s，则根据图像平面与空间平面的变换关系，可以得到以X 1，Y 1，Z 1为未知数的方程：
Figure PCTCN2022137563-appb-000002
其中，H＝MW，对该方程求解，可以得到该点在空间平面Q上的坐标(X 1，Y 1，Z 1)。
进一步地，将该点在空间平面Q上的坐标(X 1，Y 1，Z 1)转换至整车坐标系中得到该点 在整车坐标系下的三维坐标(X 2，Y 2，Z 2)。
示例性地，可以基于先验人头模型，通过透视n点(perspective-n-point，PnP)算法确定三维坐标(X 2，Y 2，Z 2)。其中，先验人头模型包括指定点的整车坐标系与图像坐标系之间的关系。具体地，某点在整车坐标系下的坐标与该点在相机坐标系下的坐标之间存在如下关系：
Figure PCTCN2022137563-appb-000003
其中，p为点在像素坐标系下的坐标，P C为点在相机坐标系下的坐标，P W为点在整车坐标系下的坐标，ω为点的深度，K为相机的内参矩阵，R CW和
Figure PCTCN2022137563-appb-000004
为从整车坐标系到相机坐标系的位姿转换，更具体地，R CW是整车坐标系到相机坐标系的旋转矩阵(将同一个向量在整车坐标系下的表示转化为在相机坐标系下的表示)，
Figure PCTCN2022137563-appb-000005
为相应的平移向量(即从相机坐标系原点指向整车坐标系原点的向量，在相机坐标系下的表示)。进一步地，根据先验人头模型中n个点在整车坐标系下的坐标，以及这n个点在图像坐标系下的坐标，能够求解R CW和
Figure PCTCN2022137563-appb-000006
进一步地，根据人脸特征点在相机坐标系下的坐标(X 1，Y 1，Z 1)可以得到该点在整车坐标系下的三维坐标(X 2，Y 2，Z 2)。
在一些可能的实现方式中，可以通过人脸关键点检测算法获取人脸部对应位置的3D位置，进而确定人脸特征点在整车坐标系下的三维坐标(X 2，Y 2，Z 2)；或者，也可以使用人头重建算法直接对用户人头进行实时重建，进而确定人脸特征点在整车坐标系下的三维坐标(X 2，Y 2，Z 2)。
应理解，人脸特征点在座舱内的三维坐标能够为显示设备调节提供更为精准的参考依据。
S404，根据第一位置信息调节显示设备的显示方向。
示例性地，本申请实施例中的显示装置为全息投影设备，其显示的影像为3D影像。示例性地，可以根据第一位置信息调节显示装置的显示的全息影像的方向，以使全息影像中包含想要传达给用户的内容的一侧面向用户，例如，当全息影像为虚拟助手的3D影像时，可以调节虚拟助手面部、或者身体和面部朝向用户，如图6中的(a)所示，当用户为副驾驶处用户时，可以调节显示虚拟助手的全息影像朝向副驾驶处用户。在一些可能的实现方式中，可以根据第一位置信息调节显示设备显示的数字人的面部朝向。
在一些可能的实现方式中，显示设备所显示的全息影像的预设方向可以如图6中的(b)所示，示例性地，全息影像为虚拟助手时，该预设方向可以为虚拟助手面部朝向的方向；全息影像为其他影像时，该预设方向可以为包含想要传达给用户的内容的一侧朝向的方向。
进一步地，根据第一位置信息调节显示设备的显示方向可以包括：调整全息影像的位姿，以使其包含想要传达给用户的内容的一侧转向用户人脸特征点的第一位置，即，将图6中(b)所示的预设方向指向人脸特征点的第一位置，具体如图6中的(c)所示。
在一些可能的实现方式中，可以通过调节全息投影设备的角度实现对全息影像位姿的调节。示例性地，全息投影设备包括投影装置和成像装置，其中，投影装置将光线投射至成像装置，以使成像装置能够显现三维立体影像。在一些可能的实现方式中，投影装置旋转时，三维立体影像发生旋转，则可以根据第一位置信息调节投影装置在垂直于车辆纵向对称平面的平面内的旋转角度，进而使全息影像包含想要传达给用户的内容的一侧朝向用户。在一些可能的实现方式中，全息投影设备的角度变换与全息影像的角度变换之间存在映射关系，例如，全息投影设备在垂直于车辆纵向对称平面的平面内顺时针旋转α度 (degree，°)，则全息影像在垂直于车辆纵向对称平面的平面内将顺时针旋转β度，其中，α和β可以相等，也可以不相等，本申请实施例对此不作限定。
示例性地，图6中的(b)所示的预设方向指示全息影像的初始位姿，记该预设方向指示全息影像的姿态角为0度，全息影像在整车坐标系中的坐标为(X3，Y3，Z3)，图6中的(c)所示的第一位置在整车坐标系中的坐标为(X4，Y4，Z4)，则根据第一位置信息和全息影像初始位姿可以确定第一旋转角度θ 1为arctan((X4-X3)/(Y4-Y3))，第一旋转方向为在垂直于车辆纵向对称平面的平面内顺时针旋转。进一步地，根据全息投影设备的角度变换与全息影像的角度变换之间的映射关系，确定全息投影设备的旋转角度θ 2，进而控制全息投影设备在垂直于车辆纵向对称平面的平面内顺时针旋转θ 2，以使全息影像包含想要传达给用户的内容的一侧朝向用户。
示例性地，上述投影装置可以实现数字高清多通道硬盘同步播放；上述成像装置可以包括分光镜。或者，上述投影装置和成像装置也可以包括其他装置，本申请实施例对此不作具体限定。
在一些可能的实现方式中，还可以根据第一位置信息调节三维立体影像的位姿。例如，在三维立体影像为虚拟助手时，可以调节虚拟助手的头部和/或身体的朝向，以使虚拟助手的头部朝向第一位置处。
在一些可能的实现方式中，还可以根据第一位置信息调节全息投影设备所显示的影像的高度。示例性地，根据第一位置信息中的高度信息，调节全息投影设备所显示的影像的高度，从而控制全息投影设备在相应的高度处显示全息影像。示例性地，可以根据第一位置中的高度信息，控制投影装置在平行于车辆纵向对称平面的平面内的旋转角度，进而全息影像的高度。
S405，获取用户的第二面部图像，根据第二面部图像确定人脸特征点的第二位置信息。
示例性地，第二位置信息用于指示人脸特征点在整车坐标系下的第二位置。
在一些可能的实现方式中，在用户与全息影像进行交互时，用户的头部位置发生了变化，此时可以获取用户的第二面部图像，并根据第二面部图像确定人脸特征点的第二位置信息。具体地，确定人脸特征点的第二位置信息的方法可以参考S403中的描述，在此不再赘述。
S406，判断第一位置和第二位置之间距离差是否大于或等于第一阈值。
具体地，在确定第一位置和第二位置之间距离差大于或等于第一阈值时，执行S407；否则，执行S408。
应理解，在大于或等于预设阈值时，触发显示设备调节，可以避免频繁调节显示设备。
示例性地，上述第一阈值可以为15厘米(centimeter，cm)，或者20cm，或者也可以为其他距离，本申请实施例对此不作具体限定。
S407，根据第二位置信息调节显示设备的显示方向。
在一些可能的实现方式中，第一位置和第二位置均为第一用户(如主驾用户)的人脸特征点在整车坐标系中的位置。则根据第二位置信息调节显示设备的显示方向可以如图6中的(d)所示。
在一些可能的实现方式中，第一位置为第一用户(如主驾驶处用户)的人脸特征点在整车坐标系中的位置，第二位置为第二用户(如副驾驶处用户)的人脸特征点在整车坐标 系中的位置。则根据第二位置信息调节显示设备的显示方向可以如图6中的(e)所示。示例性地，第二位置和第一位置指示不同用户时，第二位置的确定可以是由第二用户主动触发的。例如，第二用户通过语音指令触发，例如“全息影像，请面向我”；或者，第二用户通过相关按钮触发，例如第二用户点击“开启全息影像”的按钮。
示例性地，可以根据第一位置信息和第二位置信息确定显示设备需要旋转的角度及方向，进而控制显示设备进行旋转，以使显示设备的显示方向朝向第二位置。在一些可能的实现方式中，显示设备在垂直于车辆纵向对称平面的平面内进行旋转。
S408，结束。
应理解，上述“结束”代表结束调节显示设备的流程。
图4示出的控制显示装置的方法的步骤或操作仅为示例性说明，本申请实施例还可以执行其他操作或者图4中的各个操作的变形。此外，图4中的各个步骤可以按照与图4呈现的不同的顺序来执行，并且有可能并非要执行图4中的全部操作。示例性地，可以不执行S405至S407。示例性地，也可以跳过S402，直接执行S403。
本申请实施例提供的一种调节显示设备的方法，能够根据用户头部的特征点在整车坐标系中的三维坐标对显示设备的显示方向进行调节，从而能够精确地调节显示设备显示的影像的朝向，为用户提供更好的交互效果，提高用户的交互体验。
图7示出了本申请实施例提供的一种调节显示设备的方法，该方法700可以应用于图1所示的场景中，也可以应用于图2所示的车辆100中，该方法也可以由图3所示的系统执行。图7示出的控制显示装置的方法的步骤或操作仅为示例性说明，本申请实施例还可以执行其他操作或者图4中的各个操作的变形。该方法700包括：
S701，获取车辆内部的音频数据，确定音频数据的声源位置。
示例性地，音频信息可以是从采集的车辆内部的音频信息中排除各种无效音频信息后得到的音频信息，无效音频信息可以是音量过低的音频信息。声源位置可以是音频信息对应的声源所处的位置，声源位置可以是与基于声源定位的灯光显示设备的相对位置，也可以是具体的位置坐标，本申请实施例对此不作具体限制。
示例性地，可以基于到达时间差(time difference of arrival，TDOA)原理根据多个声音传感器采集的音频信息确定声源位置。例如，声音传感器A和B分别检测到音频从声源S处发出，其中声源S的声源信号到达声音传感器A的时间为t1，到达声音传感器B的时间为t2，则时间差dt＝|t1-t2|，设定声源S与声音传感器A的距离为AS，声源S与声音传感器B的距离为BS，音速为c，则可以得到dt＝t1-t2＝AS/c-BS/c，再根据两个声音传感器之间的距离a，选择其中一个传感器为基准点，即可确定声源的位置。
在一些可能的实现方式中，音频信息可以为包含特定唤醒词的语音，例如“开启显示设备”；或者，“虚拟助手，请面向我”；或者，也可以为其他语音信息。
S702，根据声源位置确定用户的位置。
在一些可能的实现方式中，在预设时长内，根据获取到的音频数据确定两处声源位置，例如确定声源位置分别主驾驶处和副驾驶处，则可以控制显示设备进行影像“复制”，控制复制后得到的两个影像分别面向主驾驶处用户和副驾驶处用户，如图8中的(a)所示。示例性地，该影像为虚拟助手的3D影像，则复制后的两个影像受同一指令控制。
在一些可能的实现方式中，在预设时长内，根据获取到的音频数据确定两处声源位置， 例如确定声源位置分别主驾驶处和副驾驶处，则可以控制显示设备显示两个影像，并控制上述两个影像分别面向主驾驶处用户和副驾驶处用户。
示例性地，上述预设时长可以为3秒，或者5秒，或者也可以为其他预设时长。
在一些可能的实现方式中，显示设备设置在前排中央扶手处，确定声源位置分别后排左侧座位处和后排右侧座位处时，则可以控制显示设备投射两个影像，且两个影像分别面向后排左侧座位处用户和后排右侧座位处用户。
在一些可能的实现方式中，当用户的数量大于2时，也可以控制显示设备投射多个影像，并调节该多个影像分别面向多个用户。
在一些可能的实现方式中，在根据声源位置确定用户的位置后，可以调节显示设备，以使显示设备投射的影像移动到声源位置处。示例性地，声源位置为副驾驶处，则可调节显示设备，以使影像的位置移动到副驾驶处，具体如图8中的(b)所示。
在一些可能的实现方式中，全息投影设备包括投影装置和成像装置，其中，投影装置将光线投射至成像装置，以使成像装置能够显现三维立体影像。在一些可能的实现方式中，调节成像装置的位置至副驾驶处，进而适应性调节投影装置的投射角度，进而在副驾驶处形成全息影像。
本申请实施例提供的一种调节显示设备的方法，多用户场景下，当一个用户发出语音指令，可以通过声源位置判定发出指令的用户是主驾、副驾还是后排乘客，进而调节相应的显示设备，以使显示设备投射的3D影像面向该用户，提高用户与显示设备投射的影像之间的交互性。在多用户场景下，显示设备还可以进行影像复制，进而使每个影像分别面对多个用户中的一个用户，进一步提升用户的交互体验和驾乘趣味性。
图9示出了本申请实施例提供的一种调节显示设备的方法，该方法900可以应用于图1所示的场景中，也可以应用于图2所示的车辆100中，该方法也可以由图3所示的系统执行。图9示出的控制显示装置的方法的步骤或操作仅为示例性说明，本申请实施例还可以执行其他操作或者图4中的各个操作的变形。该方法900包括：
S901，获取第一用户的第一面部图像。
具体地，获取第一用户的第一面部图像的方法，可以参考上述实施例中的描述，在此不再赘述。
S902，根据该第一面部图像确定该第一用户的人脸特征点的第一空间坐标。
在一些可能的实现方式中，上述第一空间坐标可以为该第一用户的人脸特征点在整车坐标系中的三维坐标。
示例性地，第一用户的人脸特征点可以为上述实施例中的点。进一步地，确定第一空间坐标的方法可以参考上述实施例中的描述，在此不再赘述。
S903，根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向，该影像的第一侧包含要传达给该第一用户的信息。
示例性地，该影像可以为全息影像，或者也可以为具有立体效果和景深的二位图像，本申请实施例对此不作具体限定。
具体地，根据第一空间坐标调节显示设备显示的影像的朝向的方法，可以参考上述实施例中的描述，在此不再赘述。
本申请实施例提供的一种调节显示设备的方法，能够让用户体会到与显示设备的交互， 有助于在交互层面提高用户的使用体验。
在本申请的各个实施例中，如果没有特殊说明以及逻辑冲突，各个实施例之间的术语和/或描述具有一致性、且可以相互引用，不同的实施例中的技术特征根据其内在的逻辑关系可以组合形成新的实施例。
上文中结合图4至图9详细说明了本申请实施例提供的方法。下面将结合图10和图11详细说明本申请实施例提供的装置。应理解，装置实施例的描述与方法实施例的描述相互对应，因此，未详细描述的内容可以参见上文方法实施例，为了简洁，这里不再赘述。
图10示出了本申请实施例提供的一种调节显示设备的装置2000的示意性框图，该装置2000包括获取单元2010和处理单元2020。获取单元2010可以实现相应的通信功能，处理单元2020用于进行数据处理。
可选地，该装置2000还可以包括存储单元，该存储单元可以用于存储指令和/或数据，处理单元2020可以读取存储单元中的指令和/或数据，以使得装置实现前述方法实施例。
该装置2000可以包括用于执行图4、图7、图9中的方法的单元。并且，该装置2000中的各单元和上述其他操作和/或功能分别为了实现图4、图7、图9中的方法实施例的相应流程。
其中，当该装置2000用于执行图9中的方法900时，获取单元2010可用于执行方法900中的S901，处理单元2020可用于执行方法900中的S902和S903。
该装置2000包括：获取单元2010，用于获取第一用户的第一面部图像；处理单元2020，用于根据该第一面部图像确定该第一用户的人脸特征点的第一空间坐标；根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向，该影像的第一侧包含要传达给该第一用户的信息。
可选地，该获取单元2010还用于：在根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，获取第二用户的第二面部图像；该处理单元2020还用于：根据该第二面部图像确定该第二用户的人脸特征点的第二空间坐标；根据该第二空间坐标调节该显示设备显示的该影像的第一侧的朝向。
可选地，该获取单元2010还用于：在根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，获取该第一用户的第三面部图像；该处理单元2020还用于：根据该第三面部图像确定该第一用户的人脸特征点的第三空间坐标；在该第一空间坐标与该第三空间坐标之间的距离大于或等于预设阈值时，根据该第三空间坐标调节该显示设备显示的该影像的第一侧的朝向。
可选地，该获取单元2010还用于：在该获取第一用户的第一面部图像之前，获取音频信息，该音频信息包括该第一用户的语音指令，该语音指令用于指示开启和/或调节该显示设备；该处理单元2020还用于：确定该音频信息的声源位置；根据该声源位置确定该第一用户的位置。
可选地，该音频信息还包括第二用户的语音指令，该处理单元2020还用于：控制该显示设备显示第一影像和第二影像；调节该显示设备使该第一影像的第一侧朝向该第一用户，且该第二影像的第一侧朝向该第二用户，其中，该显示设备显示的该影像包括该第一影像和/或第二影像，该第一影像的第一侧包含要传达给该第一用户的信息，该第二影像的第一侧包含要传达给该第二用户的信息。
可选地，该处理单元2020还用于：根据该第一空间坐标调节该显示设备显示的该影像的位置。
可选地，该获取单元2010还用于：在根据该第一空间坐标调节显示设备显示的影像的第一侧的朝向之前，获取该显示设备的初始位姿信息，该初始位姿信息用于指示该显示设备的姿态角和位置；该处理单元2020还用于：根据该第一空间坐标和该初始位姿信息确定该显示设备的旋转角度和旋转方向；根据该旋转角度和该旋转方向，调节该显示设备显示的该影像的第一侧的朝向。
可选地，该第一用户的人脸特征点的第一空间坐标用于指示该第一用户的人脸特征点在车辆中的三维位置，该车辆包括该显示设备。
可选地，该显示设备包括全息投影设备，该影像包括三维3D影像。
应理解，以上装置中各单元的划分仅是一种逻辑功能的划分，实际实现时可以全部或部分集成到一个物理实体上，也可以物理上分开。此外，装置中的单元可以以处理器调用软件的形式实现；例如装置包括处理器，处理器与存储器连接，存储器中存储有指令，处理器调用存储器中存储的指令，以实现以上任一种方法或实现该装置各单元的功能，其中处理器例如为通用处理器，例如CPU或微处理器，存储器为装置内的存储器或装置外的存储器。或者，装置中的单元可以以硬件电路的形式实现，可以通过对硬件电路的设计实现部分或全部单元的功能，该硬件电路可以理解为一个或多个处理器；例如，在一种实现中，该硬件电路为ASIC，通过对电路内元件逻辑关系的设计，实现以上部分或全部单元的功能；再如，在另一种实现中，该硬件电路为可以通过PLD实现，以FPGA为例，其可以包括大量逻辑门电路，通过配置文件来配置逻辑门电路之间的连接关系，从而实现以上部分或全部单元的功能。以上装置的所有单元可以全部通过处理器调用软件的形式实现，或全部通过硬件电路的形式实现，或部分通过处理器调用软件的形式实现，剩余部分通过硬件电路的形式实现。
在本申请实施例中，处理器是一种具有信号的处理能力的电路，在一种实现中，处理器可以是具有指令读取与运行能力的电路，例如CPU、微处理器、GPU、或DSP等；在另一种实现中，处理器可以通过硬件电路的逻辑关系实现一定功能，该硬件电路的逻辑关系是固定的或可以重构的，例如处理器为ASIC或PLD实现的硬件电路，例如FPGA。在可重构的硬件电路中，处理器加载配置文档，实现硬件电路配置的过程，可以理解为处理器加载指令，以实现以上部分或全部单元的功能的过程。此外，还可以是针对人工智能设计的硬件电路，其可以理解为一种ASIC，例如NPU、TPU、DPU等。
可见，以上装置中的各单元可以是被配置成实施以上方法的一个或多个处理器(或处理电路)，例如：CPU、GPU、NPU、TPU、DPU、微处理器、DSP、ASIC、FPGA，或这些处理器形式中至少两种的组合。
此外，以上装置中的各单元可以全部或部分可以集成在一起，或者可以独立实现。在一种实现中，这些单元集成在一起，以片上系统(system-on-a-chip，SOC)的形式实现。该SOC中可以包括至少一个处理器，用于实现以上任一种方法或实现该装置各单元的功能，该至少一个处理器的种类可以不同，例如包括CPU和FPGA，CPU和人工智能处理器，CPU和GPU等。
在具体实现过程中，上述获取单元2010和处理单元2020所执行的各项操作可以由同 一个处理器执行，或者，也可以由不同的处理器执行，例如分别由多个处理器执行。一示例，一个或多个处理器可以与图2中的感知系统120中一个或多个传感器相连接，从一个或多个传感器中获取用户的面部图像并进行处理；又一示例，一个或多个处理器还可以与显示装置130中的一个或多个显示设备相连接，进而控制显示设备显示的影像的朝像。示例性地，在具体实现过程中，上述一个或多个处理器可以设置在车机中的处理器，或者也可以为设置在其他车载终端中的处理器。示例性地，在具体实现过程中，上述装置2000可以为设置在车机或者其他车载终端中的芯片。示例性地，在具体实现过程中，上述装置2000可以为设置在车辆中的如图2所示的计算平台150。
图11是本申请实施例的一种调节显示设备的装置的示意性框图。图11所示的调节显示设备的装置2100可以包括：处理器2110、收发器2120以及存储器2130。其中，处理器2110、收发器2120以及存储器2130通过内部连接通路相连，该存储器2130用于存储指令，该处理器2110用于执行该存储器2130存储的指令，以收发器2120接收/发送部分参数。可选地，存储器2130既可以和处理器2110通过接口耦合，也可以和处理器2110集成在一起。
需要说明的是，上述收发器2120可以包括但不限于输入/输出接口(input/output interface)一类的收发装置，来实现装置2100与其他设备或通信网络之间的通信。
处理器2110可以采用通用的CPU，微处理器，ASIC，GPU或者一个或多个集成电路，用于执行相关程序，以实现本申请方法实施例的调节显示设备的方法。处理器2110还可以是一种集成电路芯片，具有信号的处理能力。在具体实现过程中，本申请的调节显示设备的方法的各个步骤可以通过处理器2110中的硬件的集成逻辑电路或者软件形式的指令完成。上述处理器2110还可以是通用处理器、DSP、ASIC、FPGA或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器2130，处理器2110读取存储器2130中的信息，结合其硬件执行本申请方法实施例的调节显示设备的方法。
存储器2130可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。
收发器2120使用例如但不限于收发器一类的收发装置，来实现装置2100与其他设备或通信网络之间的通信。例如，可以通过收发器2120获取用户的面部图像和/或音频信息。
本申请实施例还提供一种车辆，该车辆可以包括上述装置2000，或者上述装置2100，以及上述显示设备。其中，该显示设备可以为全息投影设备，显示的影像可以为三维影像；或者，该显示设备也可以为车载显示屏，显示的影像可以为数字人；或者，该显示设备也可以为其他显示设备，本申请对此不作具体限定。
本申请实施例还提供了一种计算机程序产品，该计算机程序产品包括：计算机程序代码，当该计算机程序代码在计算机上运行时，使得计算机执行上述方法。
本申请实施例还提供一种计算机可读存储介质，该计算机可读介质存储有程序代码，当该计算机程序代码在计算机上运行时，使得该计算机执行上述图4、图7、图9中的任一种方法。
本申请实施例还提供一种芯片，包括：至少一个处理器和存储器，该至少一个处理器与该存储器耦合，用于读取并执行该存储器中的指令，以执行上述图4、图7、图9中的任一种方法。
本申请将围绕包括多个设备、组件、模块等的系统来呈现各个方面、实施例或特征。应当理解和明白的是，各个系统可以包括另外的设备、组件、模块等，并且/或者可以并不包括结合附图讨论的所有设备、组件、模块等。此外，还可以使用这些方案的组合。
另外，在本申请实施例中，“示例的”、“例如”等词用于表示作例子、例证或说明。本申请中被描述为“示例”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用示例的一词旨在以具体方式呈现概念。
本申请实施例中，“相应的(corresponding，relevant)”和“对应的(corresponding)”有时可以混用，应当指出的是，在不强调其区别时，其所要表达的含义是一致的。
本申请实施例描述的网络架构以及业务场景是为了更加清楚地说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定，本领域普通技术人员可知，随着网络架构的演变和新业务场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
在本说明书中描述的参考“一个实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此，在本说明书中的不同之处出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是必然都参考相同的实施例，而是意味着“一个或多个但不是所有的实施例”，除非是以其他方式另外特别强调。术语“包括”、“包含”、“具有”及它们的变形都意味着“包括但不限于”，除非是以其他方式另外特别强调。
本申请中，“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：包括单独存在A，同时存在A和B，以及单独存在B的情况，其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项(个)”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b，或c中的至少一项(个)，可以表示：a，b，c，a-b，a-c，b-c，或a-b-c，其中a，b，c可以是单个，也可以是多个。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通 过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (22)

  1. 一种调节显示设备的方法，其特征在于，包括：
    获取第一用户的第一面部图像；
    根据所述第一面部图像确定所述第一用户的人脸特征点的第一空间坐标；
    根据所述第一空间坐标调节显示设备显示的影像的第一侧的朝向，所述影像的第一侧包含要传达给所述第一用户的信息。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，所述方法还包括：
    获取第二用户的第二面部图像；
    根据所述第二面部图像确定所述第二用户的人脸特征点的第二空间坐标；
    根据所述第二空间坐标调节所述显示设备显示的所述影像的第一侧的朝向。
  3. 根据权利要求1所述的方法，其特征在于，所述根据所述第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，所述方法还包括：
    获取所述第一用户的第三面部图像；
    根据所述第三面部图像确定所述第一用户的人脸特征点的第三空间坐标；
    在所述第一空间坐标与所述第三空间坐标之间的距离大于或等于预设阈值时，根据所述第三空间坐标调节所述显示设备显示的所述影像的第一侧的朝向。
  4. 根据权利要求1至3中任一项所述的方法，其特征在于，所述获取第一用户的第一面部图像之前，所述方法还包括：
    获取音频信息，所述音频信息包括所述第一用户的语音指令，所述语音指令用于指示开启和/或调节所述显示设备；
    确定所述音频信息的声源位置；
    根据所述声源位置确定所述第一用户的位置。
  5. 根据权利要求4所述的方法，其特征在于，所述音频信息还包括第二用户的语音指令，所述方法还包括：
    控制所述显示设备显示第一影像和第二影像；
    调节所述显示设备使所述第一影像的第一侧朝向所述第一用户，且所述第二影像的第一侧朝向所述第二用户，其中，所述显示设备显示的所述影像包括所述第一影像和/或第二影像，所述第一影像的第一侧包含要传达给所述第一用户的信息，所述第二影像的第一侧包含要传达给所述第二用户的信息。
  6. 根据权利要求1至5中任一项所述的方法，其特征在于，所述方法还包括：
    根据所述第一空间坐标调节所述显示设备显示的所述影像的位置。
  7. 根据权利要求1至6中任一项所述的方法，其特征在于，所述根据所述第一空间坐标调节显示设备显示的影像的第一侧的朝向之前，所述方法还包括：
    获取所述显示设备的初始位姿信息，所述初始位姿信息用于指示所述显示设备的姿态角和位置；
    所述根据所述第一空间坐标调节显示设备显示的影像的第一侧的朝向，包括：
    根据所述第一空间坐标和所述初始位姿信息确定所述显示设备的旋转角度和旋转方向；
    根据所述旋转角度和所述旋转方向，调节所述显示设备显示的所述影像的第一侧的朝向。
  8. 根据权利要求1至7中任一项所述的方法，其特征在于，所述第一用户的人脸特征点的第一空间坐标用于指示所述第一用户的人脸特征点在车辆中的三维位置，所述车辆包括所述显示设备。
  9. 根据权利要求1至8中任一项所述的方法，其特征在于，所述显示设备包括全息投影设备，所述影像包括三维3D影像。
  10. 一种调节显示设备的装置，其特征在于，包括：
    获取单元，用于获取第一用户的第一面部图像；
    处理单元，用于根据所述第一面部图像确定所述第一用户的人脸特征点的第一空间坐标；根据所述第一空间坐标调节显示设备显示的影像的第一侧的朝向，所述影像的第一侧包含要传达给所述第一用户的信息。
  11. 根据权利要求10所述的装置，其特征在于，所述获取单元还用于：
    在根据所述第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，获取第二用户的第二面部图像；
    所述处理单元还用于：根据所述第二面部图像确定所述第二用户的人脸特征点的第二空间坐标；根据所述第二空间坐标调节所述显示设备显示的所述影像的第一侧的朝向。
  12. 根据权利要求10所述的装置，其特征在于，所述获取单元还用于：
    在根据所述第一空间坐标调节显示设备显示的影像的第一侧的朝向之后，获取所述第一用户的第三面部图像；
    所述处理单元还用于：根据所述第三面部图像确定所述第一用户的人脸特征点的第三空间坐标；在所述第一空间坐标与所述第三空间坐标之间的距离大于或等于预设阈值时，根据所述第三空间坐标调节所述显示设备显示的所述影像的第一侧的朝向。
  13. 根据权利要求10至12中任一项所述的装置，其特征在于，所述获取单元还用于：
    在获取第一用户的第一面部图像之前，获取音频信息，所述音频信息包括所述第一用户的语音指令，所述语音指令用于指示开启和/或调节所述显示设备；
    所述处理单元还用于：确定所述音频信息的声源位置；根据所述声源位置确定所述第一用户的位置。
  14. 根据权利要求13所述的装置，其特征在于，所述音频信息还包括第二用户的语音指令，所述处理单元还用于：
    控制所述显示设备显示第一影像和第二影像；
    调节所述显示设备使所述第一影像的第一侧朝向所述第一用户，且所述第二影像的第一侧朝向所述第二用户，其中，所述显示设备显示的所述影像包括所述第一影像和/或第二影像，所述第一影像的第一侧包含要传达给所述第一用户的信息，所述第二影像的第一侧包含要传达给所述第二用户的信息。
  15. 根据权利要求10至14中任一项所述的装置，其特征在于，所述处理单元还用于：
    根据所述第一空间坐标调节所述显示设备显示的所述影像的位置。
  16. 根据权利要求10至15中任一项所述的装置，其特征在于，所述第一用户的人脸特征点的第一空间坐标用于指示所述第一用户的人脸特征点在车辆中的三维位置，所述车辆包括所述显示设备。
  17. 根据权利要求10至16中任一项所述的装置，其特征在于，所述获取单元还用于：在根据所述第一空间坐标调节显示设备显示的影像的第一侧的朝向之前，获取所述显示设备的初始位姿信息，所述初始位姿信息用于指示所述显示设备的姿态角和位置；
    所述处理单元还用于：根据所述第一空间坐标和所述初始位姿信息确定所述显示设备的旋转角度和旋转方向；根据所述旋转角度和所述旋转方向，调节所述显示设备显示的所述影像的第一侧的朝向。
  18. 根据权利要求10至17中任一项所述的装置，其特征在于，所述显示设备包括全息投影设备，所述影像包括三维3D影像。
  19. 一种调节显示设备的装置，其特征在于，包括：
    存储器，用于存储计算机程序；
    处理器，用于执行所述存储器中存储的计算机程序，以使得所述装置执行如权利要求1至9中任一项所述的方法。
  20. 一种车辆，其特征在于，包括权利要求10至18中任一项所述的装置和所述显示设备。
  21. 一种计算机可读存储介质，其特征在于，其上存储有计算机程序，所述计算机程序被计算机执行时，以使得实现如权利要求1至9中任一项所述的方法。
  22. 一种芯片，其特征在于，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，以执行如权利要求1至9中任一项所述的方法。
PCT/CN2022/137563 2022-05-10 2022-12-08 调节显示设备的方法和装置 WO2023216580A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210505723.1A CN115061565A (zh) 2022-05-10 2022-05-10 调节显示设备的方法和装置
CN202210505723.1 2022-05-10

Publications (1)

Publication Number Publication Date
WO2023216580A1 true WO2023216580A1 (zh) 2023-11-16

Family

ID=83198342

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/137563 WO2023216580A1 (zh) 2022-05-10 2022-12-08 调节显示设备的方法和装置

Country Status (2)

Country Link
CN (1) CN115061565A (zh)
WO (1) WO2023216580A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115061565A (zh) * 2022-05-10 2022-09-16 华为技术有限公司 调节显示设备的方法和装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111163303A (zh) * 2018-11-08 2020-05-15 中移(苏州)软件技术有限公司 一种图像显示方法、装置、终端及存储介质
CN112771480A (zh) * 2020-06-28 2021-05-07 华为技术有限公司 交互方法及电子设备
CN113467600A (zh) * 2020-03-31 2021-10-01 深圳光峰科技股份有限公司 基于增强现实的信息显示方法、系统、装置及投影设备
CN113467601A (zh) * 2020-03-31 2021-10-01 深圳光峰科技股份有限公司 基于增强现实的信息显示方法、系统、装置及投影设备
CN115061565A (zh) * 2022-05-10 2022-09-16 华为技术有限公司 调节显示设备的方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111163303A (zh) * 2018-11-08 2020-05-15 中移(苏州)软件技术有限公司 一种图像显示方法、装置、终端及存储介质
CN113467600A (zh) * 2020-03-31 2021-10-01 深圳光峰科技股份有限公司 基于增强现实的信息显示方法、系统、装置及投影设备
CN113467601A (zh) * 2020-03-31 2021-10-01 深圳光峰科技股份有限公司 基于增强现实的信息显示方法、系统、装置及投影设备
CN112771480A (zh) * 2020-06-28 2021-05-07 华为技术有限公司 交互方法及电子设备
CN115061565A (zh) * 2022-05-10 2022-09-16 华为技术有限公司 调节显示设备的方法和装置

Also Published As

Publication number Publication date
CN115061565A (zh) 2022-09-16

Similar Documents

Publication Publication Date Title
CN106575154B (zh) 全息对象的智能透明度
US10546429B2 (en) Augmented reality mirror system
CN113366491B (zh) 眼球追踪方法、装置及存储介质
WO2022241638A1 (zh) 一种投影方法及装置、车辆及ar-hud
US11682138B2 (en) Localization and mapping using images from multiple devices
WO2023216580A1 (zh) 调节显示设备的方法和装置
US11451760B2 (en) Systems and methods for correcting rolling shutter artifacts
US20200234509A1 (en) Method and computing device for interacting with autostereoscopic display, autostereoscopic display system, autostereoscopic display, and computer-readable storage medium
WO2023193482A1 (zh) 显示方法、装置、电子设备和计算机可读存储介质
US20230274455A1 (en) Systems and methods for low compute high-resolution depth map generation using low-resolution cameras
US10866425B1 (en) Image reprojection based on intra-pupil distance
US11475582B1 (en) Method and device for measuring physical objects
EP4254348A2 (en) Method and device for multi-camera hole filling
US11150470B2 (en) Inertial measurement unit signal based image reprojection
WO2019146194A1 (ja) 情報処理装置、及び情報処理方法
JP7465133B2 (ja) 情報処理装置、情報処理方法
US20240087334A1 (en) Information process system
US20230376109A1 (en) Image processing apparatus, image processing method, and storage device
CN115088015B (zh) 点云集群的空间关系
WO2023184140A1 (zh) 显示方法、装置及系统
WO2023183041A1 (en) Perspective-dependent display of surrounding environment
CN117173252A (zh) Ar-hud驾驶员眼盒标定方法、系统、设备及介质
CN117275376A (zh) 用于车载平视显示的重影消除方法、以及相关设备和芯片
CN112581628A (zh) 用于解决焦点冲突的方法和设备
CN115334296A (zh) 一种立体图像显示方法和显示装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22941515

Country of ref document: EP

Kind code of ref document: A1