WO2026026041A1 - 控制方法、装置、头戴显示设备及介质 - Google Patents

控制方法、装置、头戴显示设备及介质

Info

Publication number
WO2026026041A1
WO2026026041A1 PCT/CN2025/090571 CN2025090571W WO2026026041A1 WO 2026026041 A1 WO2026026041 A1 WO 2026026041A1 CN 2025090571 W CN2025090571 W CN 2025090571W WO 2026026041 A1 WO2026026041 A1 WO 2026026041A1
Authority
WO
WIPO (PCT)
Prior art keywords
canvas
application
information
degree
head
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/090571
Other languages
English (en)
French (fr)
Inventor
李昱锋
刘耀诚
杨明明
史高建
王文
邱绪东
张超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Goertek Inc
Original Assignee
Goertek Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Goertek Inc filed Critical Goertek Inc
Publication of WO2026026041A1 publication Critical patent/WO2026026041A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/398—Synchronisation thereof; Control thereof
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]

Definitions

  • the present invention relates to the field of head-mounted display technology, and more specifically, to a control method, a control device, a head-mounted display device, and a computer-readable storage medium.
  • One object of the present invention is to provide a control method, apparatus, head-mounted display device, and medium.
  • a control method comprising:
  • the second degree of freedom information is determined based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device and allocated to the first 3D application, so that the first canvas displays the image at the center of the first 3D application.
  • the canvas information of the first canvas is the first angle between the center position of the first canvas and the vertical positive direction of the device coordinate system of the head-mounted display device.
  • the step of determining the second degree of freedom information based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device includes:
  • the canvas information of the first canvas is the first angle between the center position of the first canvas and the vertical positive direction of the device coordinate system of the head-mounted display device.
  • the method further includes:
  • the second product between the inverse of the rotation quaternion of the first included angle and the rotation quaternion of the third degree of freedom information of the control device is obtained as the fourth degree of freedom information and assigned to the first virtual identifier corresponding to the first 3D application, so that the first virtual identifier is aligned with the direction of the second virtual identifier corresponding to the 3D desktop application.
  • the method further includes:
  • the collision position between the second virtual identifier and the rendered scene is obtained;
  • the first 3D application is controlled by the first virtual identifier to respond to the touch event.
  • the collision location is a preset location.
  • controlling the first 3D application to respond to the touch event via the first virtual identifier includes:
  • the coordinate information of the collision location is rotated in the opposite direction by the first included angle with the center of the world coordinate system as the origin, and then the first 3D application is controlled to respond to the touch event through the first virtual identifier.
  • the method further includes:
  • the step of detecting whether the first canvas among the at least one canvas is located at the center of the rendering scene is then performed.
  • a control device comprising:
  • the display module is used to display the corresponding 3D application screen in at least one canvas created in the rendering scene of the 3D desktop application.
  • a detection module is used to detect whether the first canvas in the at least one canvas is located at the center of the rendering scene; wherein the first canvas displays the screen of the first 3D application;
  • the determining module is configured to determine second degree of freedom information and allocate it to the first 3D application based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device when the first canvas is not located at the center of the rendering scene, so that the first canvas displays the image at the center of the first 3D application.
  • the canvas information of the first canvas is the first angle between the center position of the first canvas and the vertical positive direction of the device coordinate system of the head-mounted display device.
  • the determining module is specifically used for:
  • a head-mounted display device comprising:
  • Memory is used to store executable computer instructions
  • a processor configured to execute the control method described in the first aspect above, under the control of the executable computer instructions.
  • a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, perform the control method described in the first aspect above.
  • One beneficial effect of this disclosure is that, when the screen of the corresponding 3D application is displayed in at least one canvas in the rendering scene of the 3D desktop application, if it is detected that the first canvas in the at least one canvas is not located in the center position of the rendering scene of the 3D desktop application, the second degree of freedom information can be determined to be allocated to the first 3D application based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, so that the screen displayed on the first canvas is the screen at the center position of the corresponding first 3D application.
  • Figure 1 is a schematic diagram of the hardware configuration of a head-mounted display device according to an embodiment of the present disclosure
  • Figure 2 is a flowchart illustrating a control method according to an embodiment of the present disclosure
  • Figure 3 is a schematic diagram of the display area of a head-mounted display device according to an embodiment of the present disclosure
  • Figure 4 is a schematic block diagram of a control device according to an embodiment of the present disclosure.
  • Figure 5 is a schematic block diagram of a head-mounted display device according to an embodiment of the present disclosure.
  • the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.
  • directional indicators such as up, down, left, right, front, back, etc.
  • Figure 1 is a block diagram of the hardware configuration of a head-mounted display device 1000 according to an embodiment of the present disclosure.
  • the head-mounted display device 1000 can be smart glasses, which can be AR glasses, or of course other devices. This disclosure does not limit the specific devices.
  • the head-mounted display device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc.
  • the processor 1100 may include, but is not limited to, a central processing unit (CPU) or a microprocessor (MCU).
  • the memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk.
  • the interface device 1300 includes, for example, various bus interfaces, such as serial bus interfaces (including USB interfaces) and parallel bus interfaces.
  • the communication device 1400 is capable of wired or wireless communication.
  • the display device 1500 is, for example, a liquid crystal display (LCD), an LED display, or an OLED (Organic Light-Emitting Diode) display.
  • the input device 1600 includes, for example, a touchscreen, a keyboard, or a gamepad.
  • the head-mounted display device 1000 can output audio information via a speaker 1700 and can acquire audio information via a microphone 1800.
  • the head-mounted display device 1000 of the embodiments of this specification may involve only some of the devices, or may also include other devices, which is not limited herein.
  • the memory 1200 of the head-mounted display device 1000 is used to store instructions for controlling the processor 1100 to operate in order to implement or support the implementation of the control method according to any embodiment.
  • Those skilled in the art can design instructions based on the schemes disclosed in this specification. How the instructions control the processor to operate is well known in the art and will not be described in detail here.
  • the head-mounted display device shown in Figure 1 is illustrative only and is by no means intended to limit this disclosure, its application, or its use.
  • FIG. 2 illustrates a control method according to an embodiment of the present disclosure.
  • the control method can be implemented by a head-mounted display device, or by a control device independent of the head-mounted display device and the head-mounted display device together, or by a cloud server, a control device and a head-mounted display device together.
  • the head-mounted display device can be AR glasses, which can be split AR glasses or integrated AR glasses.
  • the control device can be a gamepad, a mouse, a mobile phone, etc.
  • control method of this embodiment may include the following steps S2100 to S2300:
  • Step S2100 Display the corresponding 3D application screen in at least one canvas created in the rendering scene of the 3D desktop application.
  • the desktop application of the head-mounted display device is called the AR Launcher application.
  • the AR Launcher application is a 3D application, and it starts automatically when the head-mounted display device is powered on.
  • the head-mounted display device if it receives a launch command to start a 3D application, it can open a virtual screen and run the 3D application through the AR Launcher application.
  • a new canvas corresponding to the virtual screen is created in the rendering scene of the AR Launcher application.
  • the created canvas can include a left canvas and a right canvas, with the left canvas located in the left part of the display area and the right canvas located in the right part of the display area.
  • a mask can be created for the canvas corresponding to the virtual screen, and the image of the corresponding 3D application can be filled into the mask, achieving the effect of launching other 3D applications within the AR Launcher application.
  • this module can calculate the pixel area occupied by the canvas in the rendering scene of the AR Launcher application based on the canvas information, so as to create a corresponding mask based on this pixel area.
  • the pixel area includes multiple pixels, and since the canvas typically includes a left canvas and a right canvas, the created mask usually includes two masks: one corresponding to the left canvas and one corresponding to the right canvas.
  • the above-mentioned launch command can be a touch input to the icon of the 3D application to be launched, or a ray event sent by the control device to the icon of the 3D application to be launched.
  • the control device can be a gamepad, mouse, mobile phone, or other devices, or a gesture event of the user to the icon of the 3D application to be launched.
  • the gaze state of the wearer of the head-mounted display device is acquired, and operations matching the gaze state are executed.
  • the control device generates a ray event for the icon of the 3D application to be launched.
  • the gaze position of the wearer of the head-mounted display device and the projection position of the ray event in the fused image are acquired, and operations matching the gaze position and projection position are executed.
  • eye-tracking interaction requires eye tracking.
  • the head-mounted display device captures three-dimensional information of the human eye, such as the three-dimensional coordinates of the human eye, and then launches the 3D application based on the three-dimensional information of the human eye. In other words, the accuracy of launching 3D applications can be improved by utilizing the three-dimensional information of the human eye.
  • the eye-tracking algorithm can determine the wearer's gaze position and the corresponding gaze duration in the fused image.
  • the gaze position and gaze duration can be recorded as the gaze state.
  • the control device performs different operations for different gaze states. For example, if the gaze state indicates that the gaze position does not change and the corresponding gaze duration reaches a preset length, then the wearer is determined to be in a gaze state, and an operation matching the gaze state is performed.
  • the operation matching the gaze state could be a confirmation operation.
  • the difference between the position of the ray event on the fused image and the wearer's gaze position is less than a preset threshold, it can be determined that the position of the ray event on the fused image coincides with the wearer's gaze position.
  • the operation corresponding to the gaze position and/or the ray event at the projection position is performed. For example, if the gaze position and/or the ray event at the projection position corresponds to the icon of a 3D application, then the 3D application is enabled.
  • the user launches 3D application 1.
  • the AR Launcher application opens virtual screen 1 to run 3D application 1 and creates a new canvas 1 corresponding to virtual screen 1 in the rendering scene of the AR Launcher application.
  • the canvas information of canvas 1 is then passed to the OpenXR Runtime rendering process.
  • the OpenXR Runtime rendering process determines the corresponding pixel points occupied by canvas 1 in the rendering scene of the AR Launcher application based on the canvas information of canvas 1, which are used as the corresponding pixel points of mask 1.
  • mask 1 is created (involving mask 1 corresponding to the left canvas and mask 1 corresponding to the right canvas), and the screen of 3D application 1 is filled into mask 1. This achieves the effect of opening a scene "window" of 3D application 1 in the scene of the AR Launcher application.
  • the user launches 3D application 2.
  • the AR Launcher application opens virtual screen 2 to run 3D application 2, and creates a new canvas 2 corresponding to virtual screen 2 in the rendering scene of the AR Launcher application.
  • the canvas information of canvas 2 is then passed to the OpenXR Runtime rendering process.
  • the OpenXR Runtime rendering process determines the corresponding pixel points occupied by canvas 2 in the rendering scene of the AR Launcher application based on the canvas information of canvas 2, which are used as the pixel points corresponding to mask 2.
  • mask 2 is created (involving mask 2 corresponding to the left canvas and mask 2 corresponding to the right canvas in canvas 2).
  • the image of 3D application 2 is then filled into mask 2. This achieves the effect of opening the scene “window” of 3D application 2 on top of the scene “window” effect of 3D application 1 already opened in the scene of the AR Launcher application.
  • 3D applications such as 3D Application 3 and 3D Application 4 in order to open "windows" of multiple scenes.
  • the first canvas displays the screen of the first 3D application.
  • the first canvas can be any one of at least one canvas.
  • step S2200 After performing the above step S2200 to detect whether the first canvas in the at least one canvas is located at the center of the rendering scene, proceed to:
  • Step S2300 If the device is not located at the center of the rendering scene, determine the second degree of freedom information based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, and allocate it to the first 3D application so that the first canvas displays the image at the center of the first 3D application.
  • the head-mounted display device when the head-mounted display device detects that the first canvas is not located in the center of the rendering scene of the AR Launcher application, it can determine the second degree of freedom information based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, and allocate it to the first 3D application so that the first canvas displays the image of the center position of the first 3D application.
  • the canvas information of the first canvas is the first angle between the center position of the first canvas and the positive vertical direction of the device coordinate system of the head-mounted display device. It should be noted that since the head-mounted display device uses the Unity engine for rendering, and the Unity engine uses a left-handed coordinate system, the device coordinate system of the head-mounted display device is typically a left-handed coordinate system, with the positive vertical direction being the positive X-axis direction.
  • the first degree of freedom information is the aforementioned 6DoF data.
  • step S2300 which determines the second degree of freedom information based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, may further include: obtaining the first product between the inverse of the rotation quaternion of the first included angle and the rotation quaternion of the first degree of freedom information, as the second degree of freedom information.
  • the angle 1 between the center position of canvas 1 and the positive Z-axis direction can be calculated in the rendering scene of the AR Launcher application.
  • the rotation quaternion of the 6DoF data is multiplied by the inverse of the rotation quaternion of angle 1 for each frame, and then allocated to 3D application 1.
  • the display image of canvas 1 is the image at the center position of 3D application 1, and the 6DoF display effect is normal.
  • the angle 2 between the center position of canvas 2 and the positive Z-axis direction can be calculated in the rendering scene of the AR Launcher application.
  • the rotation quaternion of the 6DoF data is multiplied by the inverse of the rotation quaternion of the angle 2 in each frame, and then allocated to 3D application 2.
  • the display image of canvas 2 is the image at the center position of 3D application 2, and the 6DoF display effect is normal.
  • the second degree of freedom information can be determined and allocated to the first 3D application based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, so that the screen displayed on the first canvas is the screen at the center position of the corresponding first 3D application.
  • control method of this embodiment further includes the following steps S3100 to S3400:
  • Step S3100 Obtain the second product between the inverse of the rotation quaternion of the first included angle and the rotation quaternion of the third degree of freedom information of the control device, and use it as the fourth degree of freedom information and assign it to the first virtual identifier corresponding to the first 3D application, so that the first virtual identifier is aligned with the direction of the second virtual identifier corresponding to the 3D desktop application.
  • the control device includes, but is not limited to, a gamepad, a mouse, and a mobile phone; however, this embodiment does not limit the scope of the control device.
  • the third degree of freedom (DOF) information refers to the rotational degree of freedom of the control device.
  • the fourth degree of freedom information is also a DOF data.
  • the first virtual identifier and the second virtual identifier can be virtual rays, which can be straight lines or curves; this embodiment does not limit this.
  • the head-mounted display device determines the second degree of freedom information and assigns it to the first 3D application based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, it can multiply the rotation quaternion of the 3DoF data of the control device by the inverse of the first included angle and use it as the fourth degree of freedom information to assign to the first virtual identifier corresponding to the first 3D application, so that the direction of the first virtual identifier is consistent with the direction of the second virtual identifier corresponding to the 3D desktop application.
  • Step S3200 In the case of assigning the fourth degree of freedom information to the first virtual identifier corresponding to the first 3D application, detect whether a touch event sent by the control device is received.
  • Touch events can be click events.
  • the head-mounted display device when the head-mounted display device assigns the recalculated 3DoF data to the first virtual identifier corresponding to the first 3D application, it can further detect whether a touch event sent by the control device has been received.
  • Step S3300 Upon detecting a touch event sent by the control device, obtain the collision position between the second virtual identifier and the rendering scene.
  • the head-mounted display device when the head-mounted display device detects a touch event sent by the control device, it can further detect the collision position between the head-mounted display device and the rendering scene of the AR Launcher application through the second virtual identifier corresponding to the AR Launcher application.
  • Step S3400 If the collision location is located on the first canvas, control the first 3D application to respond to the touch event through the first virtual identifier.
  • the first 3D application when a collision is detected at the first canvas, the first 3D application can be controlled to respond to the touch event via the first virtual identifier corresponding to the first 3D application.
  • the head-mounted display device can multiply the 3DoF data from the control device by the inverse of the rotation quaternion of the aforementioned angle 1, and assign it to virtual ray 1 so that the direction of virtual ray 1 is consistent with the direction of virtual ray 2 corresponding to the AR Launcher application.
  • the head-mounted display device can multiply the 3DoF data from the control device by the inverse of the rotation quaternion of the aforementioned angle 2, and assign it to virtual ray 3 so that the direction of virtual ray 3 is consistent with the direction of virtual ray 2 corresponding to the AR Launcher application.
  • virtual ray 2 detects the collision location between the head-mounted display and the rendered scene of the AR Launcher application. If the collision location is on canvas 1, virtual ray 1 can control 3D application 1 to respond to the touch event. If the collision location is on canvas 2, virtual ray 3 can control 3D application 2 to respond to the touch event. Conversely, virtual ray 2 can control the AR Launcher application to respond to the touch event.
  • the head-mounted display device can ensure the correct control of the control device by recalculating the 3DoF data of the control device.
  • the collision location may be a set location, which can be understood as a location where a combination of virtual and real elements is required, such as, but not limited to, locations for image recognition and locations for planar detection.
  • step S3400 where the collision location is located on the first canvas, further includes controlling the first 3D application to respond to the touch event through the first virtual identifier: where the collision location is located on the first canvas, after rotating the coordinate information of the collision location in the opposite direction by the first included angle with the center of the world coordinate system as the origin, the first 3D application is then controlled to respond to the touch event through the first virtual identifier.
  • the head-mounted display device when the collision location is located on the first canvas, can rotate the coordinate information of the collision location in the opposite direction by a first included angle with the center of the world coordinate system as the origin, and then control the first 3D application to respond to touch events through the first virtual identifier, so that the virtual combination effect is not affected.
  • the head-mounted display device will recalculate the coordinates of the touch positions that need to be combined with the virtual reality, thereby achieving correct virtual-real combination.
  • control method of this disclosure further includes the following steps S4100 to S4300:
  • Step S4100 Receive adjustment input applied to the first canvas.
  • Step S4200 In response to the adjustment input, update the position of the first canvas in the rendering scene.
  • Step S4300 After updating the position of the first canvas in the rendering scene, perform the step of detecting whether the first canvas in the at least one canvas is located at the center position of the rendering scene.
  • the AR Launcher application will transmit the adjusted first angle to the OpenXR Runtime rendering process to recalculate the 6DoF data corresponding to the first 3D application.
  • FIG 4 is a schematic diagram of a control device according to one embodiment. As shown in Figure 4, the control device 400 includes a display module 410, a detection module 420, and a determination module 430.
  • Display module 410 is used to display the screen of the corresponding 3D application in at least one canvas created in the rendering scene of the 3D desktop application.
  • the detection module 420 is used to detect whether the first canvas in the at least one canvas is located at the center of the rendering scene; wherein the first canvas displays the screen of the first 3D application;
  • the determining module 430 is configured to determine second degree of freedom information and allocate it to the first 3D application based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device when the first canvas is not located at the center of the rendering scene, so that the first canvas displays the image at the center of the first 3D application.
  • the canvas information of the first canvas is the first angle between the center position of the first canvas and the vertical positive direction of the device coordinate system of the head-mounted display device.
  • the determining module 430 is specifically used to: obtain the first product between the inverse of the rotation quaternion of the first angle and the rotation quaternion of the first degree of freedom information, as the second degree of freedom information.
  • the canvas information of the first canvas is the first angle between the center position of the first canvas and the vertical positive direction of the device coordinate system of the head-mounted display device, and the device 400 further includes an acquisition module (not shown in the figure).
  • the acquisition module is used to acquire the second product between the inverse of the rotation quaternion of the first included angle and the rotation quaternion of the third degree of freedom information of the control device, and use it as the fourth degree of freedom information and assign it to the first virtual identifier corresponding to the first 3D application, so that the first virtual identifier is aligned with the direction of the second virtual identifier corresponding to the 3D desktop application.
  • the device 400 further includes a control module (not shown).
  • the detection module 420 is further configured to detect whether a touch event sent by the control device is received when the fourth degree of freedom information is assigned to the first virtual identifier corresponding to the first 3D application.
  • the acquisition module is further configured to acquire the collision position between the second virtual identifier and the rendering scene when a touch event sent by the control device is detected;
  • a control module is configured to control the first 3D application to respond to the touch event via the first virtual identifier when the collision location is within the first canvas.
  • the collision position is a set position.
  • the control module is specifically used to rotate the coordinate information of the collision position in the opposite direction by the first included angle with the center of the world coordinate system as the origin when the collision position is located on the first canvas, and then control the first 3D application to respond to the touch event through the first virtual identifier.
  • the device 400 further includes a receiving module and an updating module (both not shown in the figure).
  • a receiving module is used to receive adjustment input applied to the first canvas
  • the update module is used to update the position of the first canvas in the rendering scene in response to the adjustment input;
  • the detection module 420 is used to detect whether the first canvas in the at least one canvas is located at the center of the rendering scene after updating the position of the first canvas in the rendering scene.
  • the second degree of freedom information can be determined and allocated to the first 3D application based on the canvas information of the first canvas and the first degree of freedom information of the head-mounted display device, so that the screen displayed on the first canvas is the screen at the center position of the corresponding first 3D application.
  • FIG. 5 is a schematic diagram of the hardware structure of a head-mounted display device according to one embodiment.
  • the head-mounted display device 500 includes a processor 510 and a memory 520.
  • the memory 520 can be used to store executable computer instructions.
  • the processor 510 can be used to execute the control method according to the method embodiments of this disclosure, under the control of the executable computer instructions.
  • the head-mounted display device 500 can be the head-mounted display device 1000 shown in Figure 1.
  • the head-mounted display device 500 may include the control device 400 described above.
  • each module of the control device 400 can be implemented by the processor 510 running computer instructions stored in the memory 520.
  • This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the control method provided in this disclosure.
  • This invention can be a system, method, and/or computer program product.
  • a computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.
  • Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device.
  • Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof.
  • Computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.
  • RAM random access memory
  • ROM read-only memory
  • EPROM or flash memory erasable programmable read-only memory
  • SRAM static random access memory
  • CD-ROM compact disc read-only memory
  • DVD digital multifunction disc
  • memory sticks floppy disks
  • mechanical encoding devices such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.
  • the computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
  • the computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing/processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and/or wireless network, to an external computer or external storage device.
  • the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing/processing device.
  • the computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages.
  • the computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.
  • the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
  • electronic circuitry such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.
  • These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions/actions specified in one or more blocks of the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and/or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions/actions specified in one or more blocks of the flowchart and/or block diagram.
  • Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions/actions specified in one or more boxes of a flowchart and/or block diagram.
  • each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function.
  • the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.
  • each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent. It should be noted that, in this document, the terms "comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • User Interface Of Digital Computer (AREA)
  • Control Of Indicators Other Than Cathode Ray Tubes (AREA)

Abstract

本发明实施例公开了一种控制方法、装置、头戴显示设备及介质，该方法包括：在3D桌面应用的渲染场景中创建的至少一个画布中分别显示对应的3D应用的画面；检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置；其中，所述第一画布显示第一3D应用的画面；在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用，以使所述第一画布显示所述第一3D应用的中心位置的画面。

Description

控制方法、装置、头戴显示设备及介质
本申请要求于2024年7月30日提交中国专利局、申请号为202411036048.8、发明名称为“控制方法、装置、头戴显示设备及介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明实施例涉及头戴显示设备技术领域，更具体地，涉及一种控制方法、一种控制装置、一种头戴显示设备、及一种计算机可读存储介质。
背景技术
在增强现实的用户体验中，在AR Launcher中进行应用多开是一种重要的使用场景。通常，在AR眼镜中实现3D应用多开时，可在AR Launcher的渲染场景中开启多个窗口，并在每个窗口内显示一个3D应用的画面。然而，当多个窗口中的任意窗口不位于AR Launcher的渲染场景的中心位置时，会导致该任意窗口内无法正常显示对应的3D应用的画面。
发明内容
本发明的一个目的是提供一种控制方法、装置、头戴显示设备及介质。
根据本发明的第一方面，提供了一种控制方法，该方法包括：
在3D桌面应用的渲染场景中创建的至少一个画布中分别显示对应的3D应用的画面；
检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置；其中，所述第一画布显示第一3D应用的画面；
在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用，以使所述第一画布显示所述第一3D应用的中心位置的画面。
可选地，所述第一画布的画布信息为所述第一画布的画布中心位置和所述头戴显示设备的设备坐标系的垂直正方向的第一夹角，
所述根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息，包括：
获取所述第一夹角的旋转四元数的逆和所述第一自由度信息的旋转四元数间的第一乘积，作为所述第二自由度信息。
可选地，所述第一画布的画布信息为所述第一画布的画布中心位置和所述头戴显示设备的设备坐标系的垂直正方向的第一夹角，
所述在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用之后，所述方法还包括：
获取所述第一夹角的旋转四元数的逆和控制设备的第三自由度信息的旋转四元数间的第二乘积，作为第四自由度信息并分配于所述第一3D应用对应的第一虚拟标识，以使所述第一虚拟标识与所述3D桌面应用对应的第二虚拟标识的方向一致。
可选地，所述方法还包括：
在将所述第四自由度信息分配于所述第一3D应用对应的第一虚拟标识的情况下，检测是否接收到所述控制设备发送的触控事件；
在检测到所述控制设备发送的触控事件的情况下，获取所述第二虚拟标识与所述渲染场景的碰撞位置；
在所述碰撞位置位于所述第一画布的情况下，通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件。
可选地，所述碰撞位置为设定位置，
所述在所述碰撞位置位于所述第一画布的情况下，通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件，包括：
在所述碰撞位置位于所述第一画布的情况下，将所述碰撞位置的坐标信息以世界坐标系的中心为原点反向旋转所述第一夹角之后，再通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件。
可选地，所述方法还包括：
接收对所述第一画布实施的调整输入；
响应于所述调整输入，更新所述第一画布在所述渲染场景中的位置；
在更新所述第一画布在所述渲染场景中的位置之后，再执行检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置的步骤。
根据本公开实施例的第二方面，提供了一种控制装置，该装置包括：
显示模块，用于在3D桌面应用的渲染场景中创建的至少一个画布中分别显示对应的3D应用的画面；
检测模块，用于检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置；其中，所述第一画布显示第一3D应用的画面；
确定模块，用于在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用，以使所述第一画布显示所述第一3D应用的中心位置的画面。
可选地，所述第一画布的画布信息为所述第一画布的画布中心位置和所述头戴显示设备的设备坐标系的垂直正方向的第一夹角，所述确定模块，具体用于：
获取所述第一夹角的旋转四元数的逆和所述第一自由度信息的旋转四元数间的第一乘积，作为所述第二自由度信息。
根据本公开实施例的第三方面，提供了一种头戴显示设备，所述头戴显示设备包括：
存储器，用于存储可执行的计算机指令；
处理器，用于根据所述可执行的计算机指令的控制，执行根据以上第一方面所述的控制方法。
根据本公开实施例的第四方面，提供了一种计算机可读存储介质，其上存储有计算机指令，所述计算机指令被处理器运行时执行以上第一方面所述的控制方法。
本公开实施例的一个有益效果在于，在3D桌面应用的渲染场景中创景的至少一个画布中分别显示对应的3D应用的画面的情况下，若检测到至少一个画布中的第一画布不位于3D桌面应用的渲染场景的中心位置，便可根据第一画布的画布信息和头戴显示设备的第一自由度信息，确定出第二自由度信息分配于第一3D应用，以使第一画布的显示画面为对应的第一3D应用的中心位置的画面。
通过以下参照附图对本说明书的示例性实施例的详细描述，本说明书的其它特征及其优点将会变得清楚。
附图说明
被结合在说明书中并构成说明书的一部分的附图示出了本发明的实施例，并且连同其说明一起用于解释本发明的原理。
图1是根据本公开实施例的头戴显示设备的硬件配置示意图；
图2是根据本公开实施例的控制方法的流程示意图；
图3是根据本公开实施例的头戴显示设备的显示区域示意图；
图4是根据本公开实施例的控制装置的原理框图；
图5是根据本公开实施例的头戴显示设备的原理框图。
本发明目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅是本发明的一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
需要说明，若本发明实施例中有涉及方向性指示(诸如上、下、左、右、前、后……)，则该方向性指示仅用于解释在某一特定姿态(如附图所示)下各部件之间的相对位置关系、运动情况等，如果该特定姿态发生改变时，则该方向性指示也相应地随之改变。
另外，若本发明实施例中有涉及“第一”、“第二”等的描述，则该“第一”、“第二”等的描述仅用于描述目的，而不能理解为指示或暗示其相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。另外，全文中出现的“和/或”的含义，包括三个并列的方案，以“A和/或B”为例，包括A方案、或B方案、或A和B同时满足的方案。另外，各个实施例之间的技术方案可以相互结合，但是必须是以本领域普通技术人员能够实现为基础，当技术方案的结合出现相互矛盾或无法实现时应当认为这种技术方案的结合不存在，也不在本发明要求的保护范围之内。
<硬件配置>
图1是根据本公开实施例的头戴显示设备1000的硬件配置的框图。
如图1所示，该头戴显示设备1000可以是智能眼镜，该智能眼镜可以是AR眼镜，当然还可以是其他设备，本公开实施例对此不作限定。
在一个实施例中，如图1所示，头戴显示设备1000可以包括处理器1100、存储器1200、接口装置1300、通信装置1400、显示装置1500、输入装置1600、扬声器1700、麦克风1800等等。
其中，处理器1100可以包括但不限于中央处理器CPU、微处理器MCU等。存储器1200例如包括ROM(只读存储器)、RAM(随机存取存储器)、诸如硬盘的非易失性存储器等。接口装置1300例如包括各种总线接口，例如串行总线接口(包括USB接口)、并行总线接口等。通信装置1400例如能够进行有线或无线通信。显示装置1500例如是液晶显示屏、LED显示屏、OLED(Organic Light-Emitting Diode)显示屏等。输入装置1600例如包括触摸屏、键盘、手柄等。头戴显示设备1000可以通过扬声器1700输出音频信息，可以通过麦克风1800采集音频信息。
本领域技术人员应当理解，尽管在图1中示出了头戴显示设备1000的多个装置，但是，本说明书实施例的头戴显示设备1000可以仅涉及其中的部分装置，也可以还包含其他装置，在此不做限定。
本实施例中，头戴显示设备1000的存储器1200用于存储指令，该指令用于控制处理器1100进行操作以实施或者支持实施根据任意实施例的控制方法。技术人员可以根据本说明书所公开方案设计指令。指令如何控制处理器进行操作，这是本领域公知，故在此不再详细描述。
在上述描述中，技术人员可以根据本公开所提供的方案设计指令。指令如何控制处理器进行操作，这是本领域公知，故在此不再详细描述。
图1所示的头戴显示设备仅是解释性的，并且决不是为了要限制本公开、其应用或用途。
下面，参照附图描述根据本公开的各个实施例和例子。
<方法实施例>
图2示出了本公开的一个实施例的控制方法，该控制方法可以由头戴显示设备实施，也可以是由独立于头戴显示设备的控制设备和头戴显示设备共同实施，还可以是由云端服务器、控制设备和头戴显示设备共同实施，其中，头戴显示设备可以是AR眼镜，该AR眼镜可以为分体式AR眼镜，也可以是一体式AR眼镜，控制设备可以是手柄、鼠标、手机等。
如图2所示，该实施例的控制方法可以包括如下步骤S2100～步骤S2300：
步骤S2100，在3D桌面应用的渲染场景中创建的至少一个画布中分别显示对应的3D应用的画面。
在本实施例中，头戴显示设备的桌面应用称之为AR Launcher应用，该AR Launcher应用为3D应用，在头戴显示设备开机时，AR Launcher应用自动启动。
在AR Launcher应用的运行过程中，若头戴显示设备接收到启动一个3D应用的启动指令，便可通过AR Launcher应用开启虚拟屏运行该3D应用，且在AR Launcher应用的渲染场景中新建虚拟屏对应的画布，通常，所创建的画布可以包括左(left)画布和右(right)画布，左画布位于显示区域的左部分，右画布位于显示区域的右部分。并且，可为虚拟屏对应的画布创建蒙版，并在蒙版中填充对应的3D应用的画面，实现在AR Launcher应用中开启其他3D应用的效果。
需要说明的是，通过在OpenXR Runtime渲染流程添加蒙版计算模块，该蒙版计算模块可用于根据上述画布的画布信息计算上述画布在AR Launcher应用的渲染场景中所占的像素区域，以便基于该像素区域创建对应的蒙版。其中，像素区域包括多个像素点，并且，由于画布通常包括左画布和右画布，所创建出的蒙版通常也包括两个蒙版，也就是说，蒙版包括与左画布对应的蒙版和右画布对应的蒙版。
需要说明的是，上述启动指令可以是针对所需开启的3D应用的图标的触摸输入，也可以是控制设备针对所需开启的3D应用的图标发送的射线事件，该控制设备可以是手柄、鼠标、手机等设备，还可以是用户针对所需开启的3D应用的图标的手势事件。
在本实施例中，在眼动交互模式下，获取头戴显示设备的佩戴者的注视状态，以及执行与注视状态相匹配的操作。控制设备针对所要开启的3D应用的图标发生射线事件，射线事件与眼动双重交互模式下，获取头戴显示设备的佩戴者的注视位置以及射线事件在融合图像中的投射位置，执行与注视位置以及投射位置相匹配的操作。通常，眼动交互需要进行眼动追踪，头戴显示设备在进行眼动追踪时，会捕捉人眼三维信息例如人眼三维坐标，进而根据人眼三维信息启动3D应用。也就是说，可利用人眼三维信息启动3D应用，这样便可提高3D应用的启动准确性。
需要说明的是，在眼动追踪模式下，可通过眼动追踪算法确定出佩戴者在融合图像上的注视位置以及注视位置对应的注视时间段。可将注视位置以及注视时间段记为注视状态。以及，在眼动追踪模式下，对于不同的注视状态，控制设备执行不同的操作。例如，在注视状态表示注视位置不发生变化，该注视位置对应的注视时间段达到预设时长的情况下，则确定佩戴者为凝视注视位置的状态，此时执行与凝视注视位置的状态相匹配的操作。例如，与凝视注视位置的状态相匹配的操作，可以为确认操作。
当然，也可在射线事件在融合图像上的位置与佩戴者的注视位置的差距小于预设阈值的情况下，确定射线事件在融合图像上的位置与佩戴者的注视位置重合，此时执行注视位置和/或射线事件在投射位置处对应的操作。例如，在注视位置和/或射线事件在投射位置处对应3D应用的图标的情况下，则开启3D应用。
示例性地，在AR Launcher应用的运行过程中，用户开启3D应用1，AR Launcher应用开启虚拟屏1运行3D应用1，并在AR Launcher应用的渲染场景中新建虚拟屏1对应的画布1，并将画布1的画布信息传递至OpenXR Runtime渲染流程。OpenXR Runtime渲染流程根据画布1的画布信息确定出画布1在AR Launcher应用的渲染场景中所占的对应像素点，作为对应蒙版1所对应的像素点。接着，参照图3，基于画布1在AR Launcher应用的渲染场景中所占的对应像素点，创建蒙版1(涉及左画布对应的蒙版1和右画布对应的蒙版1)，并将3D应用1的画面填充至蒙版1，便可实现在AR Launcher应用的场景中打开了一个3D应用1的场景“窗口”的效果。
接着，在AR Launcher应用和3D应用1的运行过程中，用户开启3D应用2，AR Launcher应用开启虚拟屏2运行3D应用2，并在AR Launcher应用的渲染场景中新建虚拟屏2对应的画布2，并将画布2的画布信息传递至OpenXR Runtime渲染流程。OpenXR Runtime渲染流程根据画布2的画布信息确定出画布2在AR Launcher应用的渲染场景中所占的对应像素点，作为蒙版2所对应的像素点。接着，参照图3，基于画布2在AR Launcher应用的渲染场景中所占的对应像素点，创建蒙版2(涉及画布2中左画布对应的蒙版2和右画布对应的蒙版2)，并将3D应用2的画面填充至蒙版2，便可实现在AR Launcher应用的场景中已经打开3D应用1的场景“窗口”效果的基础上，继续打开了3D应用2的场景“窗口”的效果。
需要说明的是，用户可继续开启3D应用3、3D应用4等3D应用，以便打开多个场景的“窗口”。
在执行以上步骤S2100在3D桌面应用的渲染场景中创建的至少一个画布中分别显示对应的3D应用的画面之后，进入：
步骤S2200，检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置。
其中，所述第一画布显示第一3D应用的画面。第一画布为至少一个画布中的任意一个画布。
在本实施例中，在头戴显示设备的AR Launcher应用的渲染场景中创建的至少一个画布中分别显示对应的3D应用的画面的情况下，头戴显示设备便可检测至少一个画布中的第一画布是否位于AR Launcher应用的渲染场景的中心位置，若第一画布位于AR Launcher应用的渲染场景的中心位置，表明第一画布的显示内容为第一3D应用的中心位置的画面。若第一画布不位于AR Launcher应用的渲染场景的中心位置，便需要通过步骤S2300重新计算第一3D应用的六自由度信息(6DoF数据)。
在执行以上步骤S2200检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置之后，进入：
步骤S2300，在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用，以使所述第一画布显示所述第一3D应用的中心位置的画面。
在本实施例中，头戴显示设备在检测到第一画布不位于AR Launcher应用的渲染场景的中心位置的情况下，便可根据第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于第一3D应用，以使第一画布显示第一3D应用的中心位置的画面。
其中，第一画布的画布信息为所述第一画布的画布中心位置和所述头戴显示设备的设备坐标系的垂直正方向的第一夹角。需要说明的是，由于头戴显示设备使用的渲染引擎为Untiy引擎，Untiy引擎使用的是左手坐标系，因此，头戴显示设备的设备坐标系通常为左手坐标系，垂直正方向为X轴正方向。
其中，第一自由度信息为上述6DoF数据。
在一个可选地实施例中，本步骤S2300根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息可以进一步包括：获取所述第一夹角的旋转四元数的逆和所述第一自由度信息的旋转四元数间的第一乘积，作为所述第二自由度信息。
以第一画布为画布1为例，头戴显示设备检测到画布1不位于AR Launcher应用的渲染场景的中心位置，此时，为了使得3D应用1在渲染场景中存在有效显示内容，便可在AR Launcher应用的渲染场景中计算画布1的画布中心位置和Z轴正方向的夹角1，并在OpenXR Runtime渲染流程中为3D应用1分配6DoF数据时，每帧画面将6DoF数据的旋转四元数乘以夹角1的旋转四元数的逆，分配给3D应用1。此时，画布1的显示画面为3D应用1的中心位置的画面，且6DoF显示效果正常。
以第一画布为画布2为例，头戴显示设备检测到画布2不位于AR Launcher应用的渲染场景的中心位置，此时，为了使得3D应用2在渲染场景中存在有效显示内容，便可在AR Launcher应用的渲染场景中计算画布2的画布中心位置和Z轴正方向的夹角2，并在OpenXR Runtime渲染流程中为3D应用2分配6DoF数据时，每帧画面将6DoF数据的旋转四元数乘以该夹角2的旋转四元数的逆，分配给3D应用2。此时，画布2的显示画面为3D应用2的中心位置的画面，且6DoF显示效果正常。
根据本公开实施例，在3D桌面应用的渲染场景中创景的至少一个画布中分别显示对应的3D应用的画面的情况下，若检测到至少一个画布中的第一画布不位于3D桌面应用的渲染场景的中心位置，便可根据第一画布的画布信息和头戴显示设备的第一自由度信息，确定出第二自由度信息分配于第一3D应用，以使第一画布的显示画面为对应的第一3D应用的中心位置的画面。
在一个实施例中，在执行以上步骤S2300在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用之后，本公开实施例的控制方法还进一步包括如下步骤S3100～步骤S3400：
步骤S3100，获取所述第一夹角的旋转四元数的逆和控制设备的第三自由度信息的旋转四元数间的第二乘积，作为第四自由度信息并分配于所述第一3D应用对应的第一虚拟标识，以使所述第一虚拟标识与所述3D桌面应用对应的第二虚拟标识的方向一致。
其中，控制设备例如但不限于包括手柄、鼠标、手机，本实施例对此不做限定。
其中，第三自由度信息为三自由度信息(简称3DoF数据)，可理解为是控制设备的旋转自由度。对应的，第四自由度信息也为三自由度信息。
其中，第一虚拟标识和第二虚拟标识可以是虚拟射线，该虚拟射线可以是直线，也可以是曲线，本实施例对此不做限定。
在本实施例中，头戴显示设备在根据第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于第一3D应用之后，便可将控制设备的3DoF数据的旋转四元数乘以上述第一夹角的逆，作为第四自由度信息分配于第一3D应用对应的第一虚拟标识，以使第一虚拟标识与3D桌面应用对应的第二虚拟标识的方向一致。
需要说明的是，对于在AR Launcher应用的渲染场景中已经开启的每个3D应用，均存在对应的虚拟标识。
步骤S3200，在将所述第四自由度信息分配于所述第一3D应用对应的第一虚拟标识的情况下，检测是否接收到所述控制设备发送的触控事件。
其中，触控事件可以是点击事件。
在本实施例中，头戴显示设备在将重新计算的3DoF数据分配于第一3D应用对应的第一虚拟标识的情况下，可进一步检测是否接收到控制设备发送的触控事件。
步骤S3300，在检测到所述控制设备发送的触控事件的情况下，获取所述第二虚拟标识与所述渲染场景的碰撞位置。
在本实施例中，头戴显示设备在检测到控制设备发送的触控事件的情况下，可进一步通过AR Launcher应用对应的第二虚拟标识检测其与AR Launcher应用的渲染场景的碰撞位置。
步骤S3400，在所述碰撞位置位于所述第一画布的情况下，通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件。
在本实施例中，在检测到碰撞位置位于第一画布的情况下，便可通过第一3D应用对应的第一虚拟标识控制第一3D应用响应该触控事件。
继续上述示例，头戴显示设备对于3D应用1对应的虚拟射线1，可将控制设备的3DoF数据乘以上述夹角1的旋转四元数的逆，分配给虚拟射线1，以使虚拟射线1的方向和AR Launcher应用对应的虚拟射线2的方向保持一致。以及，头戴显示设备对于3D应用2对应的虚拟射线3，可将控制设备的3DoF数据乘以上述夹角2的旋转四元数的逆，分配给虚拟射线3，以使虚拟射线3的方向和AR Launcher应用对应的虚拟射线2的方向保持一致。
头戴显示设备在接收到控制设备发送的触控事件的情况下，由虚拟射线2检测其与AR Launcher应用的渲染场景的碰撞位置，若碰撞位置位于画布1，便可通过虚拟射线1控制3D应用1响应该触控事件。若碰撞位置位于画布2，便可通过虚拟射线3控制3D应用2响应该触控事件。反之，便可通过虚拟射线2控制AR Launcher应用响应该触控事件。
通过本实施例，头戴显示设备通过重新计算控制设备的3DoF数据，能够保证控制设备的正确控制。
在一个实施例中，所述碰撞位置可为设定位置，设定位置可以理解为是需要做虚实结合的位置，例如但不限于包括图像识别的位置、平面检测的位置。
上述步骤S3400在所述碰撞位置位于所述第一画布的情况下，通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件可以进一步包括：在所述碰撞位置位于所述第一画布的情况下，将所述碰撞位置的坐标信息以世界坐标系的中心为原点反向旋转所述第一夹角之后，再通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件。
在本实施例中，在碰撞位置位于第一画布的情况下，头戴显示设备可将碰撞位置的坐标信息以世界坐标系的中心为原点反向旋转第一夹角之后，再通过第一虚拟标识控制所述第一3D应用响应触控事件，使得虚拟结合效果不受影响。
通过本实施例，头戴显示设备会重新计算需要进行虚实结合的触控位置的坐标，进而实现正确的虚实结合。
在一个实施例中，本公开实施例的控制方法还进一步包括如下步骤S4100～步骤S4300：
步骤S4100，接收对所述第一画布实施的调整输入。
步骤S4200，响应于所述调整输入，更新所述第一画布在所述渲染场景中的位置。
步骤S4300，在更新所述第一画布在所述渲染场景中的位置之后，再执行检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置的步骤。
通过本实施例，在用户将第一画布的位置进行调整之后，上述第一画布的画布中心位置和头戴显示设备的设备坐标系的垂直正方向的第一夹角便会自动进行调整，此时，AR Launcher应用会将调整后的第一夹角传输至OpenXR Runtime渲染流程重新进行第一3D应用对应的6DoF数据的计算。
<装置实施例>
图4是根据一个实施例的控制装置的原理示意图，参照图4所示，所述控制装置400包括显示模块410、检测模块420和确定模块430。
显示模块410，用于在3D桌面应用的渲染场景中创建的至少一个画布中分别显示对应的3D应用的画面；
检测模块420，用于检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置；其中，所述第一画布显示第一3D应用的画面；
确定模块430，用于在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用，以使所述第一画布显示所述第一3D应用的中心位置的画面。
在一个实施例中，所述第一画布的画布信息为所述第一画布的画布中心位置和所述头戴显示设备的设备坐标系的垂直正方向的第一夹角，所述确定模块430，具体用于：获取所述第一夹角的旋转四元数的逆和所述第一自由度信息的旋转四元数间的第一乘积，作为所述第二自由度信息。
在一个实施例中，所述第一画布的画布信息为所述第一画布的画布中心位置和所述头戴显示设备的设备坐标系的垂直正方向的第一夹角，装置400还包括获取模块(图中未示出)。
获取模块，用于获取所述第一夹角的旋转四元数的逆和控制设备的第三自由度信息的旋转四元数间的第二乘积，作为第四自由度信息并分配于所述第一3D应用对应的第一虚拟标识，以使所述第一虚拟标识与所述3D桌面应用对应的第二虚拟标识的方向一致。
在一个实施例中，装置400还包括控制模块(图中未示出)。
检测模块420，还用于在将所述第四自由度信息分配于所述第一3D应用对应的第一虚拟标识的情况下，检测是否接收到所述控制设备发送的触控事件；
获取模块，还用于在检测到所述控制设备发送的触控事件的情况下，获取所述第二虚拟标识与所述渲染场景的碰撞位置；
控制模块，用于在所述碰撞位置位于所述第一画布的情况下，通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件。
在一个实施例中，所述碰撞位置为设定位置，控制模块，具体用于在所述碰撞位置位于所述第一画布的情况下，将所述碰撞位置的坐标信息以世界坐标系的中心为原点反向旋转所述第一夹角之后，再通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件。
在一个实施例中，装置400还包括接收模块、更新模块(图中均未示出)。
接收模块，用于接收对所述第一画布实施的调整输入；
更新模块，用于响应于所述调整输入，更新所述第一画布在所述渲染场景中的位置；
检测模块420，用于在更新所述第一画布在所述渲染场景中的位置之后，再检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置。
根据本公开实施例，在3D桌面应用的渲染场景中创景的至少一个画布中分别显示对应的3D应用的画面的情况下，若检测到至少一个画布中的第一画布不位于3D桌面应用的渲染场景的中心位置，便可根据第一画布的画布信息和头戴显示设备的第一自由度信息，确定出第二自由度信息分配于第一3D应用，以使第一画布的显示画面为对应的第一3D应用的中心位置的画面。
<设备实施例>
图5是根据一个实施例的头戴显示设备的硬件结构示意图。如图5所示，该头戴显示设备500包括处理器510和存储器520。
该存储器520可以用于存储可执行的计算机指令。
该处理器510可以用于根据所述可执行的计算机指令的控制，执行根据本公开方法实施例所述的控制方法。
该头戴显示设备500可以是如图1所示的头戴显示设备1000。
在另外的实施例中，该头戴显示设备500可以包括以上控制装置400。
在一个实施例中，以上控制装置400的各模块可以通过处理器510运行存储器520中存储的计算机指令实现。
<计算机可读存储介质>
本公开实施例还提供了一种计算机可读存储介质，其上存储有计算机指令，所述计算机指令被处理器运行时执行本公开实施例提供的控制方法。
本发明可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本发明的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本发明操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本发明的各个方面。
这里参照根据本发明实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本发明的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本发明的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。对于本领域技术人员来说公知的是，通过硬件方式实现、通过软件方式实现以及通过软件和硬件结合的方式实现都是等价的。需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者系统不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者系统所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者系统中还存在另外的相同要素。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在如上的一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，电视机，或者网络设备等)执行本发明各个实施例的方法。
以上所述仅为本发明的优选实施例，并非因此限制本发明的专利范围，凡是在本发明的发明构思下，利用本发明说明书及附图内容所作的等效结构变换，或直接/间接运用在其他相关的技术领域均包括在本发明的专利保护范围。

Claims (10)

  1. 一种控制方法，其特征在于，所述方法包括：
    在3D桌面应用的渲染场景中创建的至少一个画布中分别显示对应的3D应用的画面；
    检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置；其中，所述第一画布显示第一3D应用的画面；
    在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用，以使所述第一画布显示所述第一3D应用的中心位置的画面。
  2. 根据权利要求1所述的方法，其特征在于，所述第一画布的画布信息为所述第一画布的画布中心位置和所述头戴显示设备的设备坐标系的垂直正方向的第一夹角，
    所述根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息，包括：
    获取所述第一夹角的旋转四元数的逆和所述第一自由度信息的旋转四元数间的第一乘积，作为所述第二自由度信息。
  3. 根据权利要求1所述的方法，其特征在于，所述第一画布的画布信息为所述第一画布的画布中心位置和所述头戴显示设备的设备坐标系的垂直正方向的第一夹角，
    所述在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用之后，所述方法还包括：
    获取所述第一夹角的旋转四元数的逆和控制设备的第三自由度信息的旋转四元数间的第二乘积，作为第四自由度信息并分配于所述第一3D应用对应的第一虚拟标识，以使所述第一虚拟标识与所述3D桌面应用对应的第二虚拟标识的方向一致。
  4. 根据权利要求3所述的方法，其特征在于，所述方法还包括：
    在将所述第四自由度信息分配于所述第一3D应用对应的第一虚拟标识的情况下，检测是否接收到所述控制设备发送的触控事件；
    在检测到所述控制设备发送的触控事件的情况下，获取所述第二虚拟标识与所述渲染场景的碰撞位置；
    在所述碰撞位置位于所述第一画布的情况下，通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件。
  5. 根据权利要求4所述的方法，其特征在于，所述碰撞位置为设定位置，
    所述在所述碰撞位置位于所述第一画布的情况下，通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件，包括：
    在所述碰撞位置位于所述第一画布的情况下，将所述碰撞位置的坐标信息以世界坐标系的中心为原点反向旋转所述第一夹角之后，再通过所述第一虚拟标识控制所述第一3D应用响应所述触控事件。
  6. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    接收对所述第一画布实施的调整输入；
    响应于所述调整输入，更新所述第一画布在所述渲染场景中的位置；
    在更新所述第一画布在所述渲染场景中的位置之后，再执行检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置的步骤。
  7. 一种控制装置，其特征在于，所述装置包括：
    显示模块，用于在3D桌面应用的渲染场景中创建的至少一个画布中分别显示对应的3D应用的画面；
    检测模块，用于检测所述至少一个画布中的第一画布是否位于所述渲染场景的中心位置；其中，所述第一画布显示第一3D应用的画面；
    确定模块，用于在不位于所述渲染场景的中心位置的情况下，根据所述第一画布的画布信息和头戴显示设备的第一自由度信息，确定第二自由度信息并分配于所述第一3D应用，以使所述第一画布显示所述第一3D应用的中心位置的画面。
  8. 根据权利要求7所述的装置，其特征在于，所述第一画布的画布信息为所述第一画布的画布中心位置和所述头戴显示设备的设备坐标系的垂直正方向的第一夹角，所述确定模块，具体用于：
    获取所述第一夹角的旋转四元数的逆和所述第一自由度信息的旋转四元数间的第一乘积，作为所述第二自由度信息。
  9. 一种头戴显示设备，其特征在于，所述头戴显示设备包括：
    存储器，用于存储可执行的计算机指令；
    处理器，用于根据所述可执行的计算机指令的控制，执行根据权利要求1-6中任意一项所述的控制方法。
  10. 一种计算机可读存储介质，其上存储有计算机指令，所述计算机指令被处理器运行时执行权利要求1-6中任意一项所述的控制方法。
PCT/CN2025/090571 2024-07-30 2025-04-23 控制方法、装置、头戴显示设备及介质 Pending WO2026026041A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202411036048.8 2024-07-30
CN202411036048.8A CN118827951A (zh) 2024-07-30 2024-07-30 控制方法、装置、头戴显示设备及介质

Publications (1)

Publication Number Publication Date
WO2026026041A1 true WO2026026041A1 (zh) 2026-02-05

Family

ID=93085347

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/090571 Pending WO2026026041A1 (zh) 2024-07-30 2025-04-23 控制方法、装置、头戴显示设备及介质

Country Status (2)

Country Link
CN (1) CN118827951A (zh)
WO (1) WO2026026041A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118827951A (zh) * 2024-07-30 2024-10-22 歌尔股份有限公司 控制方法、装置、头戴显示设备及介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111213183A (zh) * 2017-10-13 2020-05-29 三星电子株式会社 渲染三维内容的方法和装置
CN113032077A (zh) * 2021-03-29 2021-06-25 联想(北京)有限公司 头戴式设备的多任务三维效果显示方法、装置及电子设备
CN113504830A (zh) * 2021-07-22 2021-10-15 优奈柯恩(北京)科技有限公司 用于头戴式显示设备的显示方法及装置
CN117215449A (zh) * 2023-08-16 2023-12-12 歌尔科技有限公司 交互控制方法、装置、电子设备及介质
US20240045207A1 (en) * 2022-08-08 2024-02-08 Lenovo (Singapore) Pte. Ltd. Concurrent rendering of canvases for different apps as part of 3d simulation
CN118827951A (zh) * 2024-07-30 2024-10-22 歌尔股份有限公司 控制方法、装置、头戴显示设备及介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111213183A (zh) * 2017-10-13 2020-05-29 三星电子株式会社 渲染三维内容的方法和装置
CN113032077A (zh) * 2021-03-29 2021-06-25 联想(北京)有限公司 头戴式设备的多任务三维效果显示方法、装置及电子设备
CN113504830A (zh) * 2021-07-22 2021-10-15 优奈柯恩(北京)科技有限公司 用于头戴式显示设备的显示方法及装置
US20240045207A1 (en) * 2022-08-08 2024-02-08 Lenovo (Singapore) Pte. Ltd. Concurrent rendering of canvases for different apps as part of 3d simulation
CN117215449A (zh) * 2023-08-16 2023-12-12 歌尔科技有限公司 交互控制方法、装置、电子设备及介质
CN118827951A (zh) * 2024-07-30 2024-10-22 歌尔股份有限公司 控制方法、装置、头戴显示设备及介质

Also Published As

Publication number Publication date
CN118827951A (zh) 2024-10-22

Similar Documents

Publication Publication Date Title
US11057549B2 (en) Techniques for presenting video stream next to camera
CN113342697B (zh) 闪存转换层仿真测试系统及方法
US11036381B2 (en) Flexible device and operation method of flexible device
CN103631768A (zh) 协作数据编辑和处理系统
US10499164B2 (en) Presentation of audio based on source
US20250182375A1 (en) Display control method and apparatus, augmented reality head-mounted device, and medium
EP2892047A2 (en) Image data output control method and electronic device supporting the same
WO2025035950A1 (zh) 交互控制方法、装置、电子设备及介质
WO2026026041A1 (zh) 控制方法、装置、头戴显示设备及介质
US20150347364A1 (en) Highlighting input area based on user input
CN115599206A (zh) 显示控制方法、装置、头戴显示设备及介质
US10770036B2 (en) Presentation of content on left and right eye portions of headset
GB2522748A (en) Detecting pause in audible input to device
WO2026000825A1 (zh) 控制方法、装置、头戴显示设备及介质
US20110126161A1 (en) Positional effects in a three-dimensional desktop environment
US10416759B2 (en) Eye tracking laser pointer
US9990117B2 (en) Zooming and panning within a user interface
CN117575882B (zh) 顶点缓冲区的更新方法、装置及存储介质
CN117909018A (zh) 应用程序图标的显示方法及装置、存储介质、电子设备
EP3128397B1 (en) Electronic apparatus and text input method for the same
CN116360906A (zh) 交互控制方法、装置、头戴显示设备及介质
US11256410B2 (en) Automatic launch and data fill of application
CN109844692B (zh) 基于客户端压力配置呈现网络页面的网络服务器
CN115494947B (zh) 交互方法、装置、近眼显示设备和可读存储介质
KR20200106483A (ko) 화면 구성 방법, 전자 장치 및 저장 매체

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25846489

Country of ref document: EP

Kind code of ref document: A1