WO2021190421A1

WO2021190421A1 - Virtual reality-based controller light ball tracking method on and virtual reality device

Info

Publication number: WO2021190421A1
Application number: PCT/CN2021/081910
Authority: WO
Inventors: 王冉冉; 杨宇; 刘帅; 赵玉峰; 周鸣岐
Original assignee: 海信视像科技股份有限公司
Priority date: 2020-03-27
Filing date: 2021-03-19
Publication date: 2021-09-30

Abstract

Provided in the present application are a virtual reality-based controller light ball tracking method and a virtual reality device. The method comprises: according to first posture information of a previous position point of a light ball, determining second posture information of a subsequent position point adjacent to the previous position point; according to the first position information of the previous position point, the first posture information, and the second posture information, determining second position information of the subsequent position point; and according to the second position information, generating and outputting a current display position of a virtual target corresponding to a controller. The described method may improve the accuracy and precision of tracking and positioning a light ball, and may quickly track and position the light ball, improve the interaction speed between a user and a virtual reality environment, and improve the user experience.

Description

Controller photosphere tracking method based on virtual reality and virtual reality equipment

Cross-reference to related applications

This application is required to be submitted to the Chinese Patent Office on March 27, 2020, the application number is 202010230449. 2, and the application name is "Virtual reality-based controller photosphere tracking method and virtual reality equipment", and it is submitted to the Chinese Patent Office on March 31, 2020 , The application number is 202010246509.X, the application name is "Virtual reality control equipment, helmets and interactive methods" and it was submitted to the Chinese Patent Office on March 27, 2020, the application number is 202010226710.1, and the application name is "Controller tracking method and The priority of the Chinese patent application of "VR system", the entire content of which is incorporated in this application by reference.

Technical field

This application relates to the field of simulation technology, and in particular to a method for tracking a controller light ball and a virtual reality device based on virtual reality.

Background technique

With the development of science and technology, technologies such as virtual reality, augmented reality (AR), mixed reality (MR) and extended reality (XR) have developed rapidly, and they have been applied to various industries. Various industries, such as three-dimensional games, military simulation training, medical simulation surgery, etc. VR, AR, MR, and XR systems generally include helmets and controllers. By tracking and manipulating objects in the virtual world on the controller, users can interact with the surrounding environment by controlling the movement of the controller.

The controller can also be called a handle; the controller can emit a ball of light; and then it is necessary to chase the position of the ball of light to complete the positioning of the target to complete the operation of virtual reality. How to locate and track the controller is a technical problem that needs to be solved in the industry.

Summary of the invention

The present application provides a controller photosphere tracking method and virtual reality equipment based on virtual reality to solve the problem of positioning error or delay in the existing photosphere tracking technology.

In the first aspect, the present application provides a method for tracking a controller photosphere based on virtual reality. The method includes:

Determine, according to the first posture information of the previous location point of the photosphere, the second posture information of the next location point adjacent to the previous location point;

Determine the second location information of the next location point according to the first location information, the first posture information, and the second posture information of the previous location point;

According to the second position information, the current display position of the virtual target corresponding to the controller is generated and output.

In a second aspect, the present application provides a virtual reality-based controller photosphere tracking device, which includes:

The first processing unit is configured to determine the second posture information of the next location point adjacent to the previous location point according to the first posture information of the previous location point of the photosphere;

A second processing unit, configured to determine second location information of the next location point according to the first location information of the previous location point, the first posture information, and the second posture information;

The third processing unit is configured to generate and output the current display position of the virtual target corresponding to the controller according to the second position information.

In the third aspect, this application provides an electronic device, including:

At least one processor; and

A memory communicatively connected with the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the method according to any one of the first aspects .

In a fourth aspect, the present application provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method described in any one of the first aspect.

In a fifth aspect, this application provides a virtual reality device, the virtual reality device including:

A display screen, which is used to display images;

A processor, the processor is configured to:

Determining, according to the first posture information of the previous position point of the photosphere on the controller, the second posture information of the next position point adjacent to the previous position point;

Determine the second location information of the next location point according to the first location information of the previous location point, the first posture information, and the second posture information;

According to the second location information, the location of the controller is determined, and then the screen is displayed.

According to the virtual reality-based controller photoball tracking method provided in the present application, the second posture information of the next location point adjacent to the previous location point is determined according to the first posture information of the previous location point of the photosphere; The first position information, the first posture information and the second posture information of the previous position point are determined, and the second position information of the next position point is determined; according to the second position information, the current display of the virtual target corresponding to the controller is generated and output Location. Using the position information and posture information of the light ball at the previous position to predict the position information of the light ball at the next position, it effectively overcomes the problem that the image processing method is easily affected by the background color of the environment where the light ball is located, and can improve the light The accuracy and precision of the ball tracking and positioning. At the same time, because there is no need to perform image acquisition and image processing on the photosphere at the latter point, it avoids the delay and jamming caused by the image acquisition and image processing process, and can quickly focus on the light. The ball is used for tracking and positioning, which improves the interaction speed between the user and the virtual reality environment and enhances the user experience.

Description of the drawings

The drawings herein are incorporated into the specification and constitute a part of the specification, show embodiments that conform to the application, and are used together with the specification to explain the principle of the application.

FIG. 1 is a schematic flowchart of a method for tracking a photosphere of a controller based on virtual reality according to an embodiment of the application;

Figure 1a is a schematic diagram of a controller equipped with a light ball provided by an embodiment of the application;

FIG. 1b is a schematic diagram of a motion trajectory of a photosphere according to an embodiment of the application;

FIG. 2 is a schematic flowchart of another method for tracking a photosphere of a controller based on virtual reality according to an embodiment of the application;

Figure 2a is a schematic diagram of a human head doing up and down rotation around the neck according to an embodiment of the application;

FIG. 2b is a schematic diagram of a human eye doing left and right rotation around the occipital bone of the head according to an embodiment of the application;

FIG. 2c is a schematic diagram of a human arm making a rotational movement centered on the elbow according to an embodiment of the application;

FIG. 2d is a schematic diagram of the light ball moving from point J to point K according to an embodiment of the application;

3 is a schematic structural diagram of a virtual reality-based controller photosphere tracking device provided by an embodiment of the application;

FIG. 4 is a schematic structural diagram of yet another virtual reality-based controller photosphere tracking device provided by an embodiment of the application;

FIG. 5 is a schematic structural diagram of a virtual reality-based controller photosphere tracking device provided by an embodiment of the application.

Figure 6 is a schematic diagram of an application scenario provided by an embodiment of the application;

FIG. 7 is a schematic diagram of a controller provided by an embodiment of the application;

FIG. 8 is a schematic flowchart of a tracking method for a controller provided by an embodiment of the application;

FIG. 9 is a schematic flowchart of another method for tracking a controller according to an embodiment of the application;

FIG. 10 is a schematic flowchart of still another method for tracking a controller according to an embodiment of the application;

FIG. 11 is a schematic structural diagram of a tracking device for a controller provided by an embodiment of the application;

12 is a schematic structural diagram of another tracking device for a controller provided by an embodiment of the application;

FIG. 13 is a schematic diagram of the hardware structure of the tracking device of the controller provided by an embodiment of the application;

FIG. 14 is a schematic structural diagram of a VR system provided by an embodiment of the application.

FIG. 15 is a scene diagram of virtual reality interaction provided by related technologies;

FIG. 16 is a control logic diagram of a virtual reality control device provided by an embodiment of the application;

FIG. 17 is a schematic diagram of touch control provided by an example of this application;

FIG. 18 is a schematic diagram of touch control provided by another example of this application;

FIG. 19 is a control logic diagram of a virtual reality control device provided by another embodiment of the application;

FIG. 20 is a schematic diagram of a posture angle provided by an embodiment of the application;

FIG. 21 is a schematic diagram of the touch main interface in the first control mode according to an embodiment of the application;

FIG. 22 is a schematic diagram of the touch principle in the first control mode provided by an embodiment of the application; FIG.

FIG. 23 is a schematic diagram of the touch principle in the second control mode provided by an embodiment of the application; FIG.

FIG. 24 is a schematic diagram of the infrared touch structure provided by an embodiment of the application;

25 is a schematic diagram of the principle of infrared touch provided by an embodiment of the application;

FIG. 26 is a schematic diagram of the control logic of the helmet provided by an embodiment of the application;

FIG. 27 is a schematic diagram of a virtual reality interactive system provided by an embodiment of the application;

FIG. 28 is a flowchart of a virtual reality interaction method provided by an embodiment of the application;

FIG. 29 is a flowchart of a virtual reality interaction method provided by another embodiment of this application;

FIG. 30 is a block diagram of a virtual reality control device provided by an embodiment of the application;

Fig. 31 is a block diagram of a helmet provided by an embodiment of the application.

Through the above drawings, the specific embodiments of the present application have been shown, which will be described in more detail later. These drawings and text description are not intended to limit the scope of the concept of the present application in any way, but to explain the concept of the present application for those skilled in the art by referring to specific embodiments.

Detailed ways

The exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the application as detailed in the appended claims. First, explain the terms involved in this application:

Light sphere: A luminous sphere used to track and locate targets in virtual reality technology. The luminous color can be a high-saturation visible light color or infrared light, which is usually equipped on a controller.

Posture: The posture and rotation of an object in three-dimensional space, expressed by rotation matrix, Euler angles, and four elements.

Inertial sensor: A sensor mainly used to detect and measure acceleration, tilt, shock, vibration, rotation and multi-degree-of-freedom (DoF) motion. It is an important component for solving navigation, orientation and motion carrier control. Usually include "gyro", "accelerometer" and "magnetometer", as follows:

(1) The gyroscope can measure the angular velocity, and the attitude can be obtained by integrating the angular velocity, but errors will occur during the integration process. As time increases, the errors will accumulate and eventually lead to obvious attitude deviations;

(2) The accelerometer can measure the acceleration of the device, which contains gravity information. Therefore, the accelerometer data can be used to correct the attitude deviation related to the direction of gravity, that is, the accelerometer can be used to correct the angle deviation of roll and pitch;

(3) Magnetometer, the yaw angle (yaw) can be calculated from the magnetometer, and the attitude can be corrected accordingly.

Specific application scenarios of this application: With the development of virtual reality technology, virtual reality technology has been applied to production and life. Users can wear virtual reality devices to complete virtual reality operations. There will be a controller in the virtual reality device, which can also be called a handle; the controller is equipped with a light ball; furthermore, it is necessary to track the position of the light ball to complete the positioning of the target to complete the virtual reality operation. For example, the user can control the character in the virtual reality world to complete the wave motion by waving a handheld controller equipped with a light ball.

In the related art, the controller can emit visible light; the captured image can be obtained according to the visible light emitted by the controller, and then image processing is performed on the captured image to obtain the position of the photosphere, and then the target can be located.

However, in the related technology, the position of the photosphere is determined completely based on the image processing method. The image processing method is easily interfered by environmental factors and the factors of the image acquisition unit itself, which may cause the position of the photosphere to be inaccurate. , Resulting in target positioning errors or positioning delays. For example, when the color of the visible light emitted by the photosphere is red and the environment where the photosphere is located has a red background color, it may cause the image acquisition unit to be unable to accurately capture the position of the photosphere, resulting in inaccurate positioning of the photosphere. Problem: At the same time, the image acquisition device needs to distinguish between the red light emitted by the photosphere and the red background color, which results in slower positioning of the photosphere, causing stalls and delays.

The virtual reality-based controller photosphere tracking method provided in this application aims to solve the above technical problems of related technologies.

The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

Fig. 1 is a schematic flowchart of a method for tracking a photosphere of a controller based on virtual reality according to an embodiment of the application. As shown in Fig. 1, the method includes:

Step 101: According to the first posture information of the previous location point of the photosphere, determine the second posture information of the next location point adjacent to the previous location point.

In this embodiment, specifically, the execution subject of this embodiment is a terminal device or a server or a controller set on the terminal device, or other devices or devices that can execute this embodiment, and this embodiment takes the execution subject as The application software set on the terminal device is taken as an example for description. The terminal device here may be a VR device.

In virtual reality technology, luminous light balls are usually used to locate and track moving targets in a spatial range. For example, a controller with luminous light spheres held or worn by the user can be used to realize the movement or action of the user. For tracking and positioning, the position of the light ball is the user's position or the position of the user's body part wearing the light ball, and the motion trajectory of the light ball is the user's motion trajectory or the user's body part wearing the light ball. Figure 1a is a schematic diagram of a controller equipped with light balls provided by this embodiment. As shown in Figure 1a, the controller can be equipped with light balls of different colors, and light balls of different colors can represent different users or different users. Body parts.

In this embodiment, the user himself remains in place, and part of the user's body (for example, head, eyes, arms, etc.) wears a controller with a light ball and performs different actions as an example for description. When it is detected that the light ball changes from the previous position point to the next position point, it means that the user's body part wearing the light ball has also moved from the previous position point to the next position point. Exemplarily, the user can wear a controller with a light ball on his head. When the user's head rotates up and down with the neck as the center, the light ball also rotates in space with the user's head. Detecting the change in the position of the photosphere can indirectly detect the position change of the user's head when rotating; or, the user can wear a controller with a photosphere in the eyes, and when the user's eyes surround the occipital bone of the user's head When rotating left and right, the light ball also makes corresponding rotation movement in space with the user's eyes, and detecting the position change of the light ball can indirectly detect the position change of the user's eye when making a rotating movement; or, The user can wear a controller with a light ball on his arm. When the user's arm rotates around the elbow, the light ball will also rotate with the user's arm in the space, and the position change of the light ball can be detected indirectly. Changes in the position of the arm during a rotational movement.

In this embodiment, specifically, after the light ball moves from the previous position point to the next position point, both the position information and the posture information of the light ball will change. In this embodiment, the first posture information of the light ball at the previous position is used to predict the second posture information of the light ball at the next position, and there is no need to use image recognition technology to check the light ball’s position. The position information of the latter position point is identified.

The "previous position point" and "next position point" mentioned in this embodiment are two adjacent position points, which can be taken from any two adjacent position points on the trajectory of the photosphere, and are not limited to The start position point and the end position point of the trajectory of the light ball, for example, may be any two position points on the trajectory of the light ball at every predetermined interval of time dt. Wherein, the preset time dt can be set according to the requirements of the tracking accuracy of the photosphere position, for example, it can be 10ms, 20ms, and so on.

According to the first posture information of the previous position point of the photosphere, the method of determining the second posture information of the next position point adjacent to the previous position point can be a conventional method in the art, for example, posture solution algorithm. Illustratively, Fig. 1b is a schematic diagram of the trajectory of a photosphere provided by this embodiment. As shown in Fig. 1b, point A and point B are two adjacent points on the trajectory of the photosphere, and point A is the previous one. The position point, point B is the latter position point, and the posture information of the light ball at point A is Q0, then the posture information Qt of the light ball at point B can be calculated according to the following equations I and II:

Qt=Qo*Δq Formula I;

Δq=ω*dt Formula II;

Among them, ω is the rotational angular velocity, and dt is the preset time interval.

Step 102: Determine the second location information of the next location point according to the first location information, the first posture information, and the second posture information of the previous location point.

In this embodiment, specifically, according to the first posture information and the second posture information, the displacement Δl of the photosphere from the previous position point to the next position point of the two adjacent position points is calculated and determined, and then according to the previous position point The first position information and the displacement Δl of one position point are calculated and the second position information of the photosphere at the latter position point is determined.

The purpose of this embodiment is to use the first position information and first posture information of the photosphere at the previous position point of the two adjacent position points to predict the second position information of the photosphere at the latter position point, without the need to correct The second position information of the photosphere at the latter position is scanned for image recognition, which can effectively overcome the operation delay and stall failure caused by the use of image recognition technology to identify the second position information of the positioning photosphere at the latter position. And other issues.

Step 103: Generate and output the current display position of the virtual target corresponding to the controller according to the second position information.

In this embodiment, specifically, according to the second position information of the photosphere in the real space, the current display position of the virtual target corresponding to the controller is generated and output, wherein the current display position of the virtual target corresponding to the controller can be output on the VR display. The current display position of the virtual target may also output the current display position of the virtual target corresponding to the controller in the virtual reality space.

According to the second position information, the method of generating and outputting the current display position of the virtual target corresponding to the controller in the VR display and/or virtual reality space may be a conventional method in the art, which will not be repeated in this embodiment.

In this embodiment, according to the first posture information of the previous position point of the photosphere, the second posture information of the next position point adjacent to the previous position point is determined; according to the first position information of the previous position point, The first posture information and the second posture information determine the second position information of the next position point; according to the second position information, the current display position of the virtual target corresponding to the controller is generated and output. Using the position information and posture information of the light ball at the previous position point to predict the position information of the light ball at the next position point, it effectively overcomes the problem that the image processing method is easily affected by the background color of the environment where the light ball is located, and can improve the light The accuracy and precision of the ball tracking and positioning. At the same time, because there is no need to perform image acquisition and image processing on the photosphere at the latter point, it avoids the delay and jamming caused by the image acquisition and image processing process, and can quickly focus on the light. The ball is used for tracking and positioning, which improves the interaction speed between the user and the virtual reality environment and enhances the user experience.

FIG. 2 is a schematic flowchart of another method for tracking a photosphere of a controller based on virtual reality according to an embodiment of the application. As shown in FIG. 2, the method includes:

Step 201: Acquire first position information and first posture information of the previous position point of the photosphere.

Camera and image recognition technology can be used to obtain the first position information of the previous position of the photosphere, specifically: the camera is used to obtain the image data when the photosphere is at the previous position, and the image recognition technology is used to compare the acquired image The data is recognized and processed, the position of the center of the photosphere is obtained, and the position of the center of the photosphere is converted into three-dimensional coordinates to obtain the first position information of the photosphere. The image recognition technology is a conventional technology in the field, and will not be repeated in this embodiment.

The inertial sensor IMU can be used to collect the IMU data at the previous position of the photosphere, and the collected IMU data can be processed to obtain the first posture information of the photosphere. Among them, the collected IMU data can be processed by using a posture calculation algorithm to obtain the first posture information of the photosphere. The first posture information of the photosphere includes at least the rotational angular velocity, acceleration, or yaw angle. Exemplarily, the inertial sensor IMU may be used to collect the gravitational acceleration when the photosphere is at the previous position point, and the rotation angular velocity can be obtained according to the gravitational acceleration.

Optionally, acquiring the first position information of the previous position of the photosphere includes: acquiring an image, where the image is the image collected by the collection unit when the photosphere is located at the previous position; and determining that the photosphere is in the image according to the image In the location to get the first location information. Among them, the acquisition unit may be a camera. In order to obtain the position information of the photosphere in the three-dimensional space, multiple cameras can be set up to collect images of the photosphere at the same time, and then the spatial triangulation algorithm is used to determine the position of the photosphere at the previous position The first location information. Before using the camera to collect the photosphere image to obtain the position information of the photosphere, the position and posture of the camera need to be calibrated in advance by using markers of known position and posture.

Optionally, obtaining the first posture information of the previous position of the photosphere includes: using a gyroscope to obtain the angular velocity of the photosphere at the previous position; using an accelerometer to obtain the acceleration of the photosphere at the previous position; using magnetism The meter obtains the yaw angle of the light ball at the previous position point. The above-mentioned methods for acquiring the first posture information by using inertial sensors may all be conventional methods in the field, and details are not described herein again in this embodiment.

Optionally, this embodiment further includes an operation of storing the acquired first location information. Store the first location information for use in subsequent steps.

Step 202: Obtain the posture data detected by the inertial measurement unit; determine the second posture information according to the first posture information, posture data, and preset movement time, where the movement time is the movement of the photosphere from the previous position to the next The time required for the location point.

In this embodiment, specifically, the inertial measurement unit includes an inertial sensor, and the attitude data includes any one of the following: rotation angular velocity, gravitational acceleration, yaw angle, and pitch angle. This embodiment uses the attitude data as the rotation angular velocity for description. .

According to the first posture information, posture data and preset movement time, determining the second posture information includes: determining the movement angle according to the posture data and the movement time; and determining the second posture information according to the movement angle and the first posture information. Among them, the movement time refers to the time required for the light ball to move from the previous position to the next position. The length of the movement time can be set according to actual needs. For example, it can be set according to the actual demand for the accuracy of the tracking and positioning of the light ball. Set the movement time. When the tracking and positioning accuracy of the light ball is required, a shorter movement time can be set. On the contrary, when the tracking and positioning accuracy of the light ball is low, a longer movement time can be set. In general, the moving time can be set to 10ms-20ms. The moving angle refers to the angle that the light ball moves in a rotating motion within the moving time.

Exemplarily, this embodiment uses the posture data as the rotational angular velocity ω for description, and the movement time is preset to dt, and the movement angle is Δq=ω*dt; assuming that the first posture information of the photosphere at the previous position is Q0 , Then the second posture information Qt of the photosphere at the latter position point is Qt=Qo*Δq.

Step 203: Determine the first predicted position when the photosphere is located at the previous position point according to the first posture information, where the first predicted position represents the position of the photosphere relative to the initial position point when the photosphere is located at the previous position point.

In this embodiment, specifically, determining the first predicted position when the photosphere is located at the previous position point according to the first posture information includes: determining the first predicted position according to the first posture information and a preset bone and joint model , Among them, the bone joint model is used to indicate the movement relationship of the human joints.

Specifically, the bone joint model is used to indicate the changes in the position or movement trajectory of the human joints over time. When a photosphere is worn at the human joints, the bone joint model can also be used to indicate the change in the position or movement trajectory of the photosphere over time. Condition. The bone joint model includes a preset moving radius; determining the first predicted position according to the first posture information and the preset bone joint model includes: determining according to the first posture information, the moving radius, and the preset first moving time The first predicted position, where the first movement time is the time required for the photosphere to move from the initial position point to the previous position point.

The bone joint model in this embodiment is adapted to the position of the human body joint, and different human joints correspond to different bone joint models. Illustratively, the bone and joint models in this embodiment include a head model, an eye model, and an arm model. In this embodiment, the human head, eyes, and arms in a two-dimensional plane xoy coordinate system are taken as examples to illustrate the bone joint model.

Figure 2a is a schematic diagram of a human head doing up and down rotation with the neck as the center provided by this embodiment. As shown in Figure 2a, the O ₁ point represents the position of the human neck, and the L point, M point and N point represent The position of the human head; the human head rotates _{from point L to point N through point M at a rotational angular velocity ω 1} , point L is the starting position of the rotation, point M and point N are two adjacent positions respectively The previous position point is the previous position point and the next position point; the distance r ₁ between the human head and the human neck is the radius of the rotational motion trajectory. Use the method of this embodiment to determine the first predicted position M(x _M , y _M ) when the photosphere is at point M:

θ _M ＝ω ₁ *dt ₁ formula (1)

x _M ＝r ₁ *sinθ _M ＝r ₁ *sin(ω ₁ *dt ₁ ) Equation (2)

y _M ＝r ₁ *cosθ _M ＝r ₁ *cos(ω ₁ *dt ₁ ) Equation (3)

The above equations (1)-(3) are the head model of this embodiment, where ω ₁ is the first posture information of the human head at point M, r ₁ is the moving radius, and dt ₁ is the preset The first move time.

Figure 2b is a schematic diagram of a human eye doing left and right rotation around the occipital bone of the head provided by this embodiment. As shown in Figure 2b, the O ₂ point represents the position of the occipital bone of the head, point F, point G and point H Represents the position of the human eye; the human eye rotates _{from point F through point G to point H at an angular velocity of rotation ω 2} , point F is the starting position of the rotational movement, point G and H are two adjacent positions respectively The previous location point and the next location point in, the distance r ₂ between the eyes of the human body and the occipital bone of the head is the radius of the rotational motion trajectory. Use the method of this embodiment to determine the first predicted position G(x _G , y _G ) when the photosphere is at point G:

θ _G ＝ω ₂ *dt ₂ formula (4)

The above equations (4)-(5) are the eye model of this embodiment, where ω ₂ is the first posture information of the human eye at point G, r ₂ is the moving radius, and dt ₂ is the preset The first move time.

Figure 2c is a schematic diagram of a human arm performing a rotational movement centered on the elbow provided by this embodiment. As shown in Figure 2c, _{point O 3} represents the position of the elbow, and point C, D and E represent the position of the human arm. Is the position; the human arm rotates _{from point C through point D to point E at an angular velocity of rotation ω 3} , point C is the starting position of the rotation movement, point D and point E are the previous positions of the two adjacent positions respectively Point and the next position point; the distance r ₃ between the human arm and the elbow is the radius of the rotational motion trajectory. Use the method of this embodiment to determine the first predicted position D(x _D , y _D ) when the photosphere is at point D:

θ _D =ω ₃ *dt ₃ formula (7)

x _D =r ₃ *sin(θ _D +β)=r ₃ *sin(ω ₃ *dt ₃ +β) Equation (8)

y _D =r ₃ *cos(θ _D +β)=r ₃ *cos(ω ₃ *dt ₃ +β) Equation (9)

The above equations (7)-(9) are the arm model of this embodiment, where ω ₃ is the first posture information of the human arm at point D, r ₃ is the moving radius, and dt ₃ is the preset first posture information. The moving time, β is the angle between the line between the starting position point C and the elbow position O _{3 and the vertical direction.}

The above equations (1)-equation (3), equation (4)-equation (5) and equation (7)-equation (9) are only for the human head, eyes and arms in the two-dimensional plane xoy coordinate system. The bone and joint model is illustrated as an example. The above method in this embodiment can also be used to determine the bone and joint models of other parts of the human body, such as the wrist model of the human wrist, which will not be repeated in this embodiment.

For the bone joint model of the human joint in the three-dimensional xoyz coordinate system, the position of the human joint in the three-dimensional xoyz coordinate system can be disassembled into the position in the two-dimensional plane xoy coordinate system, xoz coordinate system, and yoz coordinate system. The above method is used to determine the bone joint models of the human joints in the above three two-dimensional planes, and then the three bone joint models are combined to obtain the bone joint models of the human joints in the three-dimensional xoyz coordinate system. In this embodiment, formula (10) is used to comprehensively express the bone joint model of the human joint:

p=f(q)=q*(0,ln,0)*q ^-1 formula (10)

Among them, p is the position information of the human joints, q is the posture information of the human joints at a certain position, q ^-1 is the inverse of q quaternion form, and ln is the moving radius of the human joints.

Step 204: Determine, according to the second posture information, a second predicted position when the light ball is located at the next position point, where the second predicted position represents the position of the light ball relative to the initial position point when the light ball is located at the next position point.

In this embodiment, specifically, determining the second predicted position when the photosphere is at the next position point according to the second posture information includes: determining the second predicted position according to the second posture information and the bone and joint model.

Determining the second predicted position according to the second posture information and the bone joint model includes: determining the second predicted position according to the second posture information, the moving radius, and the preset second moving time, where the second moving time is a photosphere The time required to move from the initial point to the next point.

For the same human joint, the bone joint model does not change with the movement of the human joint, that is to say, when determining the first prediction model and the second prediction model of the photosphere at two adjacent positions The bone and joint model used is the same.

The method and principle of step 204 are similar to or the same as the method and principle of step 203, please refer to the related record of step 203, which will not be repeated here.

Step 205: Determine the movement displacement of the light ball according to the second predicted position and the first predicted position, where the movement displacement represents the displacement of the light ball from the previous position point to the next position point.

In this embodiment, specifically, according to the second predicted position and the first predicted position, the distance between the second predicted position and the first predicted position is calculated in the spatial coordinate system, that is, the light sphere moves from the previous position to the The displacement of the latter position point.

Illustratively, Fig. 2d is a schematic diagram of the light ball moving from point J to point K provided by this embodiment. As shown in Fig. 2d, point J and point K are the previous one and For the latter position point, use the method of this embodiment to calculate the displacement of the photosphere from point J to point K:

_{The first predicted position p J} and the second predicted position p _K when the photosphere is at point J and point K are determined by formula (10) as:

p _J = f(q _J ) = q _J *(0,ln,0)*q _J ^-1

p _K = f(q _K ) = q _K *(0,ln,0)*q _K ^-1

Then, the displacement Δl of the photosphere from point J to point K is:

Δl=p _K -p _J =q _K *(0,ln,0)*q _K ^-1 -q _J *(0,ln,0)*q _J ^-1

The above-mentioned method is only used for the explanation of this embodiment, and is not used to limit this application. This application may also use other methods to determine the movement and displacement of the photosphere, which will not be repeated in this embodiment.

Step 206: Determine second position information according to the movement displacement and the first position information; according to the second position information, generate and output the current display position of the virtual target corresponding to the controller.

In this embodiment, specifically, the first position information is the real position information of the light ball at the previous position point. The first position information of the light ball at the previous position point and the movement displacement are comprehensively superimposed to obtain the light The second position information of the ball in the latter position.

Exemplarily, in Figure 2d, assuming that the first position information of the photosphere at point J is p, the method of this embodiment is used to determine the second position information p _t of the photosphere at point K as:

p _t =p+Δl=p+q _K *(0,ln,0)*q _K ^-1 -q _J *(0,ln,0)*q _J ^-1

The above-mentioned method is only used for the explanation of this embodiment, and is not used to limit this application. This application may also use other methods to determine the second position information of the photosphere, which will not be repeated in this embodiment.

Optionally, before generating and outputting the current display position of the virtual target corresponding to the controller according to the second position information, the method further includes: smoothing the second position information according to the pre-stored position information of the historical position point of the photosphere After processing, the smoothed second position information is obtained.

Smoothing the second position information can reduce the noise or distortion of the image. The method of smoothing the second position information in this embodiment may be a conventional method in the art, such as mean filtering and median filtering. Method, Gaussian filtering method or bilateral filtering method, etc.

Optionally, the method of the present application further includes: generating and outputting the current pose information of the photosphere according to the second position information and the second posture information. Among them, the pose information includes position information and pose information, and the current pose information of the photosphere is generated and output, so that the pose information can be cited when the photosphere is continuously tracked and positioned.

In this embodiment, the first position information and the first posture information of the previous position of the photosphere are acquired; the posture data detected by the inertial measurement unit is obtained; according to the first posture information, the posture data, and the preset moving time , Determine the second posture information, where the movement time is the time required for the light ball to move from the previous position point to the next position point; according to the first posture information, determine the first predicted position when the light ball is at the previous position point, Among them, the first predicted position represents the position of the photosphere relative to the initial position point when the photosphere is located at the previous location point; according to the second posture information, the second predicted position when the photosphere is located at the next location point is determined, where the second prediction The position characterizes the position of the light ball relative to the initial position point when it is located at the next position point; according to the second predicted position and the first predicted position, the movement displacement of the light ball is determined, where the movement displacement characterizes the movement of the light ball from the previous position point to The displacement of the latter position point; the second position information is determined according to the movement displacement and the first position information; the current display position of the virtual target corresponding to the controller is generated and output according to the second position information. Using the position information and posture information of the light ball at the previous position to predict the position information of the light ball at the next position, it effectively overcomes the problem that the image processing method is easily affected by the background color of the environment where the light ball is located, and can improve the light The accuracy and precision of the ball tracking and positioning. At the same time, because there is no need to perform image acquisition and image processing on the photosphere at the latter position, it avoids problems such as delays and freezes caused by the image acquisition and image processing process, and can quickly focus on the light. The ball is used for tracking and positioning, which improves the interaction speed between the user and the virtual reality environment and improves the user experience; further, the first predicted position of the light ball at the previous location point and the second predicted location at the next location point are used to determine The movement displacement of the light ball from the previous position point to the next position point, and then according to the actually measured first position information of the light ball at the previous position point and the movement displacement, determine the second position of the light ball at the next position point. The location information can further improve the accuracy and precision of the photosphere tracking and positioning.

Fig. 3 is a schematic structural diagram of a virtual reality-based controller photosphere tracking device provided by an embodiment of the application. As shown in Fig. 3, the device includes:

The first processing unit 1 is configured to determine the second posture information of the next location point adjacent to the previous location point according to the first posture information of the previous location point of the photosphere;

The second processing unit 2 is configured to determine the second location information of the next location point according to the first location information, the first posture information, and the second posture information of the previous location point;

The third processing unit 3 is configured to generate and output the current display position of the virtual target corresponding to the controller according to the second position information.

Fig. 4 is a schematic structural diagram of another virtual reality-based controller photosphere tracking device provided by an embodiment of the application. On the basis of Fig. 3, as shown in Fig. 4:

The second processing unit 2 includes:

The first processing subunit 21 is configured to determine, according to the first posture information, a first predicted position when the light ball is located at the previous position point, where the first predicted position represents that when the light ball is located at the previous position point, relative to the initial position Point location

The second processing subunit 22 is configured to determine a second predicted position when the light ball is located at the next position point according to the second posture information, where the second predicted position represents that when the light ball is located at the next position point, relative to the initial position Point location

The third processing subunit 23 is configured to determine the movement displacement of the photosphere according to the second predicted position and the first predicted position, where the movement displacement represents the displacement of the photosphere from the previous position point to the next position point;

The fourth processing subunit 24 is configured to determine the second position information according to the movement displacement and the first position information.

The first processing subunit 21 includes:

The first processing module 211 is configured to determine a first predicted position according to the first posture information and a preset bone joint model, where the bone joint model is used to indicate the movement relationship of the human joints;

The second processing subunit 22 includes:

The second processing module 221 is configured to determine the second predicted position according to the second posture information and the bone joint model.

Wherein, the bone joint model includes a preset moving radius; the first processing module 211 includes:

The first processing sub-module 2111 is used to determine the first predicted position according to the first posture information, the movement radius, and the preset first movement time, where the first movement time is the movement of the photosphere from the initial position point to the previous position Point required time;

The second processing module 221 includes:

The second processing sub-module 2211 is used to determine the second predicted position according to the second posture information, the movement radius, and the preset second movement time, where the second movement time is the movement of the photosphere from the initial position point to the next position Point the time required.

The first processing unit 1, including:

The fifth processing subunit 11 is used to obtain the posture data detected by the inertial measurement unit;

The sixth processing subunit 12 is used to determine the second posture information according to the first posture information, posture data, and preset movement time, where the movement time is required for the photosphere to move from the previous position point to the next position point time.

The sixth processing subunit 12 includes:

The third processing module 121 is configured to determine the movement angle according to the posture data and the movement time;

The fourth processing module 122 is configured to determine the second posture information according to the movement angle and the first posture information.

Among them, the attitude data is any one of the following: rotation angular velocity, gravitational acceleration, yaw angle, and pitch angle.

The device also includes an acquiring unit 4, which is used for before the first processing unit 1 determines the second posture information of the next location point adjacent to the previous location point according to the first posture information of the previous location point of the photosphere, Acquiring the first position information and the first posture information of the previous position point of the photosphere;

Among them, the obtaining unit 4 includes:

The acquiring subunit 41 is used to acquire an image, where the image is an image acquired by the acquisition unit when the photosphere is located at a previous position;

The seventh processing subunit 42 is used to determine the position of the light ball in the image according to the image to obtain the first position information.

The device also includes:

The fourth processing unit 5 is configured to perform a correction based on the pre-stored position information of the historical position of the photosphere before the third processing unit 3 generates and outputs the current display position of the virtual target corresponding to the controller according to the second position information. Smoothing is performed on the second position information to obtain smoothed second position information.

The device also includes:

The fifth processing unit 6 is configured to generate and output the current pose information of the photosphere according to the second position information and the second posture information.

According to the embodiments of the present application, the present application also provides an electronic device and a readable storage medium.

As shown in FIG. 5, it is a block diagram of an electronic device based on a virtual reality-based controller photosphere tracking method according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the application described and/or required herein.

As shown in FIG. 5, the electronic device includes: one or more processors 501, a memory 502, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. The various components are connected to each other using different buses, and can be installed on a common motherboard or installed in other ways as needed. The processor may process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input/output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and/or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). In FIG. 5, a processor 501 is taken as an example.

The memory 502 is a non-transitory computer-readable storage medium provided by this application. The memory stores instructions that can be executed by at least one processor, so that the at least one processor executes the virtual reality-based controller photosphere tracking method provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, and the computer instructions are used to make the computer execute the virtual reality-based controller photosphere tracking method provided by the present application.

As a non-transitory computer-readable storage medium, the memory 502 can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program corresponding to the virtual reality-based controller photosphere tracking method in the embodiment of the present application Instructions/modules (for example, the acquisition unit 1, the first processing unit 2, and the second processing unit 3 shown in FIG. 3). The processor 501 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 502, that is, realizing the virtual reality-based controller photosphere tracking method in the above method embodiment .

The memory 502 may include a storage program area and a storage data area. The storage program area may store an operating system and an application program required by at least one function; Created data, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 502 may optionally include memories remotely provided with respect to the processor 501, and these remote memories may be connected to an electronic device based on virtual reality-based photosphere tracking via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device based on the virtual reality-based photosphere tracking method may further include: an input device 503 and an output device 504. The processor 501, the memory 502, the input device 503, and the output device 504 may be connected by a bus or in other ways. In FIG. 5, the connection by a bus is taken as an example.

The input device 503 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device based on virtual reality photosphere tracking, such as touch screen, keypad, mouse, track pad, touch pad , Pointing stick, one or more mouse buttons, trackball, joystick and other input devices. The output device 504 may include a display device, an auxiliary lighting device (for example, LED), a tactile feedback device (for example, a vibration motor), and the like. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, the programmable processor It can be a dedicated or general-purpose programmable processor that can receive data and instructions from the storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device. An output device.

These calculation programs (also referred to as programs, software, software applications, or codes) include machine instructions for programmable processors, and can be implemented using high-level procedures and/or object-oriented programming languages, and/or assembly/machine language Calculation program. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and/or device used to provide machine instructions and/or data to a programmable processor ( For example, magnetic disks, optical disks, memory, programmable logic devices (PLD)), including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

In order to provide interaction with the user, the systems and techniques described here can be implemented on a computer that has: a display device for displaying information to the user (for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) ); and a keyboard and a pointing device (for example, a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (for example, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, voice input, or tactile input) to receive input from the user.

The systems and technologies described herein can be implemented in a computing system that includes back-end components (for example, as a data server), or a computing system that includes middleware components (for example, an application server), or a computing system that includes front-end components (for example, A user computer with a graphical user interface or a web browser, through which the user can interact with the implementation of the system and technology described herein), or includes such back-end components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be connected to each other through any form or medium of digital data communication (for example, a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

The computer system can include clients and servers. The client and server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs that run on the corresponding computers and have a client-server relationship with each other.

In related technologies, the controller carries an inertial measurement unit (IMU). The IMU can measure the angular velocity and acceleration of the controller in a three-dimensional space, and use this to calculate the controller’s attitude to achieve three degrees of freedom ( Three Degrees of Freedom, 3DOF for short) tracking.

However, in the above-mentioned technology, the position of the controller cannot be measured, and the degree of freedom of movement along the three rectangular coordinate axes X, Y, and Z cannot be obtained. Therefore, it is difficult to track the position change of the controller when the user manipulates the controller for translation. , Resulting in poor interaction between users and the surrounding environment, affecting user experience.

In the embodiment of the present application, a multi-point light-emitting unit is set on the controller, and multiple light points of the controller are tracked by a visual method, so as to track the position and posture of the controller, thereby achieving 6DOF tracking of the controller.

This embodiment provides a tracking method for a controller. The method can be applied to the application scenario diagram shown in FIG. 6. As shown in FIG. 6, the application scenario provided by this embodiment includes the tracking processor 101 of the controller and the control The device 102 and the image acquisition device 103. Fig. 7 is a schematic diagram of a controller provided by this embodiment. As shown in Fig. 7, the controller carries a multi-point light-emitting unit, and the multi-point light-emitting unit includes multiple light points. The restriction can be set according to actual application scenarios. For example, a possible use form of the controller is a handle, and the user can hold the handle with his hand and control the movement of the handle. The tracking processor 101 of the controller can obtain the converted sequence images of the multi-point light-emitting unit during the movement of the controller 102 according to the image obtaining device 103, and then track the position and posture of the controller, and determine the controller Six degrees of freedom tracking data.

The above application scenario is only an exemplary scenario. During specific implementation, it can be applied in different scenarios according to requirements. For example, the application scenario includes a tracking processor and an image acquisition device, as well as any one of a bracelet, a ring, and a watch. The bracelet, ring, or watch carries a multi-point light-emitting unit, and the multi-point light-emitting unit includes a plurality of light points, so as to realize the tracking of the bracelet, ring or watch.

In the following, specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

FIG. 8 is a schematic flowchart of a tracking method for a controller provided by an embodiment of the application. The controller carries a multi-point light-emitting unit. The execution subject of this embodiment may be the controller in the embodiment shown in FIG. 6 Trace the processor. As shown in Figure 8, the method may include:

S301: Obtain, according to the image acquisition device, the transformed sequence image of the multi-point light-emitting unit during the movement of the controller, and determine the transformation mode of the light points in the sequence image.

Exemplarily, the aforementioned image acquisition device may be a monocular, binocular, or multi-lens camera that comes with the all-in-one machine. When the image acquisition device is a binocular or multi-lens camera, each camera independently acquires the controller during the movement process. Transformation sequence image of multi-point lighting unit. Binocular or multi-eye cameras can expand the tracking range, but this embodiment is also applicable to monocular cameras. Take a monocular camera as an example. The camera captures an image of a multi-point light-emitting unit during the movement of the controller. The multi-point light-emitting unit includes multiple light points. In this embodiment, the number of light points is greater than or equal to 4. The number can be set according to the actual application scenario. Each light point is transformed according to a different transformation method. This embodiment does not limit the transformation method. For example, the first light point transformation method is RGBRGB; the second light point transformation The mode is RRGGBB; the third light point conversion mode is 101010; the fourth light point conversion mode is 110011, where RGB represents red, green, and blue respectively, and 1, 0 represents light and dark respectively. It can be understood that the color conversion is not limited to red, green and blue, and can be red, orange, yellow, green, blue, blue, and purple; the brightness level is not limited to full bright and full dark, and can be multiple brightness levels, such as full brightness, 3/4 brightness, and half brightness. Bright, 1/4 bright, dark, etc., the conversion method can also include color conversion and brightness level conversion at the same time. The conversion sequence image of the multi-point light-emitting unit is acquired by the image acquisition device, and the conversion mode of the light point in the sequence image can be determined according to the information such as the color and brightness level of the light spots in the sequence image. Among them, this embodiment does not limit the implementation of obtaining information such as the color and brightness level of the light spots in the sequence image. For example, the difference threshold of the color value of each preset color can be set according to the actual situation. If the difference between the color value and the color value of a preset color is less than the first preset difference threshold, then the color of the light spot is the preset color; similarly, the brightness level of each brightness level can be set according to the actual situation. The difference threshold of the spot diameter. If the difference between the brightness value of the spot and the spot diameter of a certain brightness level is less than the second preset difference threshold, the brightness of the spot is the brightness level.

The change frequency of the above-mentioned multi-point light-emitting unit can be set according to the actual application scenario. The shooting frequency of the image acquisition device should be consistent with the change frequency of the multi-point light-emitting unit, so that the shooting of the image acquisition device is synchronized with the conversion of the multi-point light-emitting unit. The image acquisition device can precisely capture the transformation of each lamp in the multi-point light-emitting unit, so that the transformation mode of the light points in the sequence image can be accurately determined.

S302: Obtain an identifier corresponding to the target light point in the sequence image according to the conversion mode of the light point.

Exemplarily, the above-mentioned multi-point light-emitting unit includes a plurality of light points, and each light point is transformed according to a different transformation method. In this way, the corresponding light point in the sequence image can be determined according to the transformation method of the light point in the sequence image. Of the logo. Among them, the number of target light points can be set according to actual application scenarios. For example, when the number of light points of the multi-point light-emitting unit is small, the number of target light points can be all light points of the multi-point light-emitting unit; When the number of light spots is large, the number of target light spots may be part of the light spots in the multi-point light-emitting unit. The selection of the target light point can also be set according to actual application scenarios, for example, during the movement of the controller, a light point that is always in a place that can be photographed by the image acquisition device.

S303: Based on the identifier corresponding to the target light spot, determine the mapping position of the target light spot in each frame of the image sequence.

Exemplarily, through the Open Source Computer Vision Library (OpenCV) to extract the target light point in the above sequence image, and obtain the horizontal and vertical pixel coordinates of the target light point, so as to obtain the light point corresponding to each logo. The mapping position in each frame of the above sequence of images.

S304: Obtain six-degree-of-freedom tracking data of the controller according to the mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller.

Input the above-mentioned mapping position, the initial position of the target light spot and the position of the image acquisition device during the movement of the controller into OpenCV, then the position of the target light spot during the movement of the controller can be obtained, and the position and posture of the controller can be determined .

The tracking method of the controller provided by the application embodiment, the controller carries a multi-point light-emitting unit, and the method acquires the transformed sequence image of the above-mentioned multi-point light-emitting unit during the movement of the controller through an image acquisition device, and determines that the sequence image is The light point transformation method; here, the transformation method of each light point is different, so the embodiment of the present application can accurately determine the identification corresponding to the target light point in the sequence image according to the light point transformation method; and then based on the above-mentioned target light point correspondence To determine the mapping position of the target light spot in each frame of the image sequence; according to the above mapping position and the initial position of the target light spot, the target light spot relative to the image acquisition device during the movement of the controller is obtained According to the position and the position of the image acquisition device during the movement of the controller, the position of the target light spot is obtained. Since the three-dimensional geometric structure of the multi-point light-emitting unit in the controller is unchanged, the target light spot is obtained The position of the controller can determine the three-dimensional space position and rotation attitude of the controller, realize the six-degree-of-freedom tracking of the controller, and improve the interaction between the user and the surrounding environment. At the same time, the tracking method of the controller provided in the embodiments of the present application does not require installation of additional devices, such as laser detection devices required for laser positioning, etc., thereby saving cost and space.

In addition, in order to solve the problem of unsmooth and delayed tracking data, the embodiment of the present application also considers the result of attitude tracking of the controller sent by the IMU. As shown in FIG. 9, FIG. 9 is a schematic flowchart of another tracking method of a controller provided by an embodiment of the application. The controller carries a multi-point light-emitting unit. The execution subject of this embodiment may be as shown in FIG. 6 The tracking processor of the controller in the embodiment. As shown in Figure 9, the method includes:

S401: Acquire, according to an image acquisition device, a transformed sequence image of the multi-point light-emitting unit during the movement of the controller, and determine a transformation mode of light points in the sequence image.

S402: Obtain an identifier corresponding to the target light point in the sequence image according to the conversion mode of the light point.

S403: Based on the identifier corresponding to the target light spot, determine the mapping position of the target light spot in each frame of the image sequence.

The implementation of S401-S403 is the same as the foregoing S301-S303, and will not be repeated here.

S404: Obtain the attitude tracking result of the controller sent by the IMU.

In this embodiment, according to the mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller, it is only possible to determine the position and posture of the controller when each frame of image is taken, resulting in tracking problems. The data is not smooth, and the above tracking method has the problem of delay. The update rate of IMU attitude tracking is fast, the delay is lower, and smooth tracking data can be obtained.

Based on this, this embodiment needs to obtain the posture tracking result of the controller sent by the IMU.

This embodiment does not limit the sequence of S404 and S401-S403, that is, S404 may be executed first, and then S401-S403, or S401-S403 may be executed first, and then S404 may be executed.

After S404, perform the following steps: obtain the six-degree-of-freedom tracking data of the controller according to the mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller .

Optionally, obtaining the six-degree-of-freedom tracking data of the controller according to the mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller includes:

S4051: Obtain the position and posture of the controller according to the mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller.

Input the above-mentioned mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller into OpenCV, then the position of the target light spot during the movement of the controller can be obtained, so as to determine the image capture in each frame When, the position and attitude of the controller.

Optionally, the number of the target light points is not less than a preset number;

The obtaining the position and posture of the controller according to the mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller includes;

Obtaining the position of the target light spot relative to the image acquisition device through a PnP algorithm according to the mapping position and the initial position of the target light spot;

Obtaining the position of the target light spot according to the position of the target light spot relative to the image acquisition device and the position of the image acquisition device during the movement of the controller;

According to the position of the target light spot, the position and posture of the controller are obtained.

Exemplarily, the number of the above-mentioned target light points is not less than the preset number, and the preset number can be set according to actual application scenarios. In this embodiment, the number of target light points is at least 4 before the PnP algorithm can be passed. , To obtain the position of the target light spot relative to the image acquisition device. After inputting the above-mentioned mapping position and the initial position of the above-mentioned target light spot into OpenCV, the position of the target light spot relative to the image acquisition device during the movement of the controller can be obtained through the PnP algorithm, and then according to the relative position of the target light spot during the movement The position of the image acquisition device and the position of the image acquisition device can obtain the position of the target light spot. Among them, the PnP algorithm is a method for solving 3D to 2D point-pair motion. It describes how to get the pose of the camera when n (n≥4) 3D space points and their mapping positions are known. The camera pose and n 3D spaces The position of the point is a relative relationship. Therefore, when the pose of the camera and the mapping position of the n 3D space points are known, the position of the n 3D space points can be obtained through the PnP algorithm. Since the three-dimensional geometric structure of the multi-point light-emitting unit in the controller is unchanged, the position of the target light point is obtained, and the three-dimensional space position and rotation attitude of the controller can be determined, thereby obtaining the six-degree-of-freedom tracking data of the controller .

It can be understood that when the above-mentioned image acquisition device is a binocular or multi-lens camera, the target light is obtained through the PnP algorithm according to the mapping position of the target light point in each frame of the image sequence of each camera and the initial position of the target light point. The position of the point relative to each camera, and based on the position of the target light point relative to each camera and the position of each image acquisition device during the movement of the controller, the positions of two or more sets of target light points are obtained. The positions of multiple groups of target light points are summed or weighted to obtain the position of the target light point, thereby improving the accuracy of obtaining the position of the target light point.

S4052: Fuse the position and posture of the controller and the result of posture tracking of the controller sent by the IMU to obtain six-degree-of-freedom tracking data of the controller.

Exemplarily, the position and posture of the controller and the result of posture tracking of the controller sent by the IMU are input into OpenCV, and mutual compensation, correction, smoothing, and prediction are performed through a preset fusion algorithm to obtain the controller’s Six degrees of freedom tracking data.

In this embodiment, by fusing the position and posture of the controller and the result of posture tracking of the controller sent by the IMU, the advantages of the update rate and smoothness of the posture tracking of the IMU can be fully utilized, and the advantages of The drift and error accumulation in IMU attitude tracking make it difficult to track the position change of the controller, and the problem of 6DOF tracking cannot be achieved. At the same time, it also solves the problem of image acquisition according to the mapping position, the initial position of the target light spot and the controller during the movement. The location of the device causes the tracking data to be unsmooth and delayed.

In addition, in the embodiment of the present application, the controller carries a multi-point light-emitting unit, and the image acquisition device acquires the transformed sequence images of the above-mentioned multi-point light-emitting unit during the movement of the controller, and determines the transformation mode of the light points in the sequence image; here The conversion mode of each light spot is different, so the embodiment of the present application can accurately determine the identification of the target light spot in the sequence image according to the conversion mode of the light spot; and then determine the target light spot based on the identification corresponding to the above-mentioned target light spot The mapping position in each frame of the image in the above sequence of images; according to the mapping position and the initial position of the target light point, the position of the target light point relative to the image acquisition device during the movement of the controller is obtained, and then according to the moving During the process, the position of the target light spot relative to the position of the image acquisition device and the position of the image acquisition device is obtained. Since the three-dimensional geometric structure of the multi-point light-emitting unit in the controller is unchanged, the target light is obtained. The position and posture of the controller can be determined by the position of the point; by fusing the position and posture of the controller and the result of the posture tracking of the controller sent by the IMU, the update rate of the posture tracking of the IMU can be fully utilized. The advantages of smoothness and other aspects solve the problem of unsmooth and delayed tracking data caused by the mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller.

FIG. 10 is a schematic flowchart of another tracking method for a controller provided by an embodiment of the application. The controller carries a multi-point light-emitting unit. The execution subject of this embodiment may be the controller in the embodiment shown in FIG. 6 Tracking processor. As shown in Figure 10, the method includes:

S501: Extract light points in the sequence of images.

S502: Based on the light points, identify the same points of adjacent frames in the sequence of images.

Optionally, the identifying the same points of adjacent frames in the sequence of images based on the light points includes:

Obtaining the distance between the centers of light spots of adjacent frames in the sequence of images;

According to the distance and a preset distance threshold, the same points of adjacent frames in the sequence of images are identified.

Exemplarily, extract the target light point in the above sequence image through OpenCV, and obtain the horizontal and vertical pixel coordinates of the target light point, and the distance between the light point centers of adjacent frames

Among them, u ₁ and v ₁ are the horizontal and vertical pixel coordinates of the light spot in the previous frame of image, u ₂ , v ₂ are the horizontal and vertical pixel coordinates of the light spot in the next frame of image respectively, and the preset distance threshold is d ₀ , If d ₁ ≤ d ₀ , it is judged that this light spot is the same point; otherwise, d ₁ >d ₀ , it is judged that the two light spots are not the same point.

Through S503: it is judged whether the same points are continuous.

If the same points are continuous, S5041 is executed, and if the same points are not continuous, S5042-S5043 are executed.

Exemplarily, when the controller is moving, sometimes a light spot in the lamp group is turned to a place that cannot be captured by the image acquisition device, and the image captured at this time does not have the light spot (the light spot in the previous frame of image) There is no such light spot in the next frame of image); sometimes there is a light spot in the light group that could not be taken before, and then reappears (the light spot is not in the latter frame of image, and the light spot in the next frame of image There is the light spot). In this case, the same point cannot be found in some images in the sequence image, and the same point is not continuous in the sequence image. On the contrary, the light point of the lamp group has not been transferred to a place that cannot be captured by the image acquisition device. The same point can be found in each frame of the sequence image, and the same point is continuous in the sequence image.

S5041: Obtain an identifier corresponding to the target light point in the sequence image according to the initial identifier of the light point.

If the same point is continuous, that is, the light point can be found in each frame of the sequence image, then the light point corresponding to the light point in each frame of the sequence image can be directly determined according to the initial identification of the light point. It is not necessary to obtain the mark corresponding to the target light point in the sequence image through the light point conversion method, which simplifies the operation process and improves the tracking efficiency.

S5042: Obtain, according to the image acquisition device, the transformed sequence image of the multi-point light-emitting unit during the movement of the controller, and determine the transformation mode of the light points in the sequence image.

Exemplarily, if the same point is not continuous in the sequence image, then the same point before and after the disconnection cannot be found in the sequence image, and then it is impossible to determine the same point in the sequence image according to the initial identification of the light point. The mark corresponding to the light spot in the image after the light spot is disconnected. Therefore, for the light spot that was not photographed before and reappears, when it reappears, it is necessary to obtain the sequence image through the light spot transformation method. The identifier corresponding to the target light spot.

Optionally, the light spot is an LED light spot, and the conversion method includes color conversion and/or brightness level conversion,

The determination of the transformation mode of the light points in the sequence image may be implemented in the following manners:

According to the color and/or brightness level of the same point, determine the color transformation and/or brightness level transformation of the same point in a set of sequence images.

Exemplarily, when the same point is discontinuous in the sequence image, when it reappears, based on the color and/or brightness level of the same point, determine the color transformation and/or brightness of the same point in a set of sequence images Level conversion, in which the number of frames of a group of sequential images is related to the conversion period of the light spot, for example, the conversion period of the light spot is four times a period, then a group of sequential images is a continuous four-frame image.

Optionally, the light spot is an infrared light spot, and the conversion method includes infrared light-dark level conversion,

The determination of the transformation mode of the light points in the sequence image may also be implemented in the following manner:

According to the infrared light-dark level of the same point, the infrared light-dark level transformation of the same point in a set of sequence images is obtained.

Exemplarily, when the same point is discontinuous in the sequence image, when it reappears, based on the infrared light and dark level of the same point, determine the infrared light and dark level transformation of the same point in a set of sequence images, one For the number of frames of the group sequence image, refer to the above-mentioned embodiment, which will not be repeated here.

S5043: Obtain an identifier corresponding to the target light point in the sequence image according to the conversion mode of the light point.

Optionally, the obtaining the identifier corresponding to the target light point in the sequence image according to the light point transformation mode includes:

According to the conversion mode of the light spot and the correspondence between the preset conversion mode and the light spot identification, the identification corresponding to the target light spot in the sequence image is obtained.

Exemplarily, based on the corresponding relationship between the preset conversion method and the light spot identifier, a preset conversion method that is the same as the color conversion method of the same point in a set of sequence images is found, and the identifier corresponding to the preset conversion method is These similarities correspond to the identification. For example, the color of the same point in a set of sequence images is transformed into RGBRGB, and the corresponding identification of the preset transformation method RGBRGB is the first light point, then the same point is the first light point; the color transformation of the same point in a set of serial images It is RRGGBB, and the identifier corresponding to the preset transformation method RRGGBB is the second light spot, then the same point is the second light spot; the brightness level of the same point in a set of sequence images is transformed to 101010, and the identifier corresponding to the preset transformation method 101010 If it is the third light point, then the same point is the third light point, where RGB represents red, green and blue respectively, and 1, 0 represents light and dark respectively.

According to the conversion mode of the light spot and the corresponding relationship between the preset conversion mode and the light spot identification, the identification corresponding to the target light spot in the sequence image can be determined more accurately and conveniently.

S505: Based on the identifier corresponding to the target light spot, determine the mapping position of the target light spot in each frame of the image sequence.

S506: Obtain six-degree-of-freedom tracking data of the controller according to the mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller.

The implementation of S505-S506 is the same as the above-mentioned S303-S304, and will not be repeated here.

The tracking method of the controller provided by the embodiment of the present application, the controller carries a multi-point light-emitting unit, the method extracts light points in a sequence image, and based on the light points, identifies the same point in adjacent frames in the sequence image, if The same point is continuous, that is, for a light point, the light point can be found in each frame of the sequence image, then the light point in each frame of the sequence image can be directly determined according to the initial identification of the light point The corresponding mark does not need to use the light point transformation method to obtain the mark corresponding to the target light point in the sequence image, which simplifies the operation process and improves the tracking efficiency; if the same point is not continuous in the sequence image, it is required when it reappears The identification of the target light spot in the sequence image is determined by the light point transformation method; the image acquisition device is used to acquire the transformed sequence image of the multi-point light-emitting unit during the movement of the controller, and the light point transformation method in the sequence image is determined ; The transformation method of each light spot is different, so according to the transformation method of the light spot and the corresponding relationship between the preset transformation method and the light spot identification, the identification of the target light spot in the sequence image can be determined more accurately and conveniently; Then, based on the identifier corresponding to the target light point, the mapping position of the target light point in each frame of the image sequence is determined; according to the mapping position and the initial position of the target light point, the target light during the movement of the controller is obtained. The position of the point relative to the image acquisition device, and then according to the position of the target light spot relative to the image acquisition device and the position of the image acquisition device during the movement, the position of the target light spot is obtained, due to the multi-point light emission in the controller The three-dimensional geometric structure of the unit is unchanged. Once the position of the target light spot is obtained, the three-dimensional space position and rotation posture of the controller can be determined, the six-degree-of-freedom tracking of the controller is realized, and the interaction between the user and the surrounding environment is improved. sex.

Corresponding to the tracking method of the controller in the above embodiment, FIG. 11 is a schematic structural diagram of a tracking device for a controller provided in an embodiment of the application. For ease of description, only the parts related to the embodiments of the present application are shown. As shown in FIG. 11, the tracking device 60 of the controller includes: a first determining module 601, a first obtaining module 602, a second determining module 603, and a second obtaining module 604.

The first determining module 601 is configured to acquire, according to the image acquisition device, the transformed sequence image of the multi-point light-emitting unit during the movement of the controller, and determine the transformation mode of the light points in the sequence image;

The first obtaining module 602 is configured to obtain the identifier corresponding to the target light point in the sequence image according to the conversion mode of the light point;

The second determining module 603 is configured to determine the mapping position of the target light point in each frame of the image sequence based on the identifier corresponding to the target light point;

The second obtaining module 604 is configured to obtain the six-degree-of-freedom tracking data of the controller according to the mapping position, the initial position of the target light spot, and the position of the image obtaining device during the movement of the controller .

The device provided in the embodiment of the present application can be used to implement the technical solutions of the foregoing method embodiments, and its implementation principles and technical effects are similar, and the details of the embodiments of the present application are not repeated here.

FIG. 12 is a schematic structural diagram of another tracking device for a controller provided by an embodiment of the application. As shown in FIG. 12, the tracking device 60 of the controller provided in this embodiment, on the basis of the embodiment in FIG. 11, further includes: an acquisition module 605 and a processing module 606.

Optionally, the obtaining module 605 is configured to, before the second obtaining module 604 obtains the six-degree-of-freedom tracking data of the controller,

Acquiring the result of the attitude tracking of the controller sent by the IMU;

The second obtaining module 604 obtains the six-degree-of-freedom tracking data of the controller, including:

Obtaining the position and posture of the controller according to the mapping position, the initial position of the target light spot, and the position of the image acquisition device during the movement of the controller;

The position and posture of the controller and the result of posture tracking of the controller sent by the IMU are merged to obtain the six-degree-of-freedom tracking data of the controller.

Optionally, the processing module 606 is configured to extract light points in the sequence of images;

Based on the light points, identifying the same points of adjacent frames in the sequence of images;

Judge whether the same points are continuous;

If the same points are continuous, the first obtaining module 602 obtains the identifier corresponding to the target light point in the sequence image according to the initial identifier of the light point;

If the same points are not continuous, the first determining module 601 executes the acquisition of the transformed sequence image of the multi-point light-emitting unit during the movement of the controller according to the image acquisition device, and determines the position of the light point in the sequence image. Steps to change the way.

The first determining module 601 determines the transformation mode of light points in the sequence image, including:

Optionally, the processing module 606 recognizes the same points of adjacent frames in the sequence of images based on the light points, including:

Optionally, the first obtaining module 602 obtains the identifier corresponding to the target light point in the sequence image according to the conversion mode of the light point, including:

The second obtaining module 604 obtains the position and posture of the controller according to the mapping position, the initial position of the target light spot, and the position of the image obtaining device during the movement of the controller, including;

FIG. 13 is a schematic diagram of the hardware structure of the tracking device of the controller provided by an embodiment of the application. As shown in FIG. 13, the tracking device 80 of the controller in this embodiment includes: a processor 801 and a memory 802; wherein

The memory 802 is used to store computer execution instructions;

The processor 801 is configured to execute computer-executable instructions stored in the memory to implement each step of the tracking method of the controller in the foregoing embodiment. For details, please refer to the relevant description in the foregoing method embodiment.

Optionally, the memory 802 may be independent or integrated with the processor 801.

When the memory 802 is set independently, the tracking device further includes a bus 803 for connecting the memory 802 and the processor 801.

The embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executed instructions, the tracking method of the controller as described above is implemented.

FIG. 14 is a schematic structural diagram of a VR system provided by an embodiment of the application. As shown in FIG. 14, the VR system 90 of this embodiment includes: an all-in-one machine 901 and a controller 902. Among them, the all-in-one machine 901 is provided with a tracking processor 9011 of a controller, an image acquisition device 9012 and the like. The controller 902 carries a multi-point light-emitting unit, and the multi-point light-emitting unit includes a plurality of light points. A possible use form of the controller is a handle. The tracking processor 9011 of the controller is configured to perform the above-mentioned method. Optionally, an IMU9021 is provided on the aforementioned controller. The implementation principles and technical effects of the VR system provided in this embodiment can be seen in the foregoing method embodiments, and will not be repeated here.

An embodiment of the present application also provides an AR system, including: an all-in-one machine and a controller. Among them, the all-in-one machine is provided with a tracking processor of the controller, an image acquisition device, and the like. The controller carries a multi-point light-emitting unit, the multi-point light-emitting unit includes a plurality of light points, and a possible use form of the controller is a handle. Optionally, an IMU is provided on the aforementioned controller. The tracking processor of the controller can be used to execute the technical solutions of the foregoing method embodiments, and its implementation principles and technical effects are similar, and the details are not repeated here in the embodiments of the present application.

The embodiment of the present application also provides an MR system, including: an all-in-one machine and a controller. Among them, the all-in-one machine is provided with a tracking processor of the controller, an image acquisition device, and the like. The controller carries a multi-point light-emitting unit, the multi-point light-emitting unit includes a plurality of light points, and a possible use form of the controller is a handle. Optionally, an IMU is provided on the aforementioned controller. The tracking processor of the controller can be used to execute the technical solutions of the foregoing method embodiments, and its implementation principles and technical effects are similar, and the details are not repeated here in the embodiments of the present application.

An embodiment of the present application also provides an XR system, including: an all-in-one machine and a controller. Among them, the all-in-one machine is provided with a tracking processor of the controller, an image acquisition device, and the like. The controller carries a multi-point light-emitting unit, and the multi-point light-emitting unit includes multiple light points. A possible use form of the controller is a handle. Optionally, an IMU is provided on the aforementioned controller. The tracking processor of the controller can be used to execute the technical solutions of the foregoing method embodiments, and its implementation principles and technical effects are similar, and the details are not repeated here in the embodiments of the present application.

Virtual reality helmet VR display has a wide range of applications in various industries such as education and training, fire drills, virtual driving, real estate and other projects. In VR scenarios, users often need to input some information to achieve VR interaction. For example, enter the account number and password; or adjust the volume and screen size while watching audio and video. At present, the relevant interactive methods in the VR field mainly include:

1) Interaction is through the touchpad that comes with VR, but because it is blindly operated when wearing a VR helmet, the position of the touchpad is not easy to determine. Moreover, the user needs to raise his arm to operate, and it will feel very tired after a long time. It is especially inconvenient when you wear more clothes in winter.

2) Head movement hovering, the input mode of the shell-type virtual reality head-mounted display device mostly depends on the inertial sensing unit in the mobile phone. The user controls the cursor position movement by turning the head, and when the cursor moves to the option to be selected (Such as confirm, return, music, video) hovering for a certain period of time (such as 3s or 5s) as a selection confirmation operation, although this method is simple, sometimes there is no response and the length of time is not well defined. If the time definition is too short, it is easy to cause misoperation. If the time definition is too long, the operation efficiency is low and it is easy to cause misunderstanding, disgust and impatientness of the user, and the user experience is very poor.

3) Speech recognition. This method can be simple and effective to achieve interaction, but sometimes ambiguity occurs, and the effect of speech semantic recognition is poor, especially when the humanities of the region are different, and the dialects and standard dialects of each place may have certain differences. Moreover, this method is not suitable for deaf-mute people and has certain limitations.

4) With the help of traditional peripherals such as keyboard and mouse, this is the most common way for VR devices to input text. However, the keyboard peripherals need to be carried by the user at any time. Users increase the burden and affect the experience.

5) Through the traditional binocular recognition gesture technology interaction, this method can be regarded as a better interaction method, but the gesture operation has certain restrictions, the interaction process has strong pertinence, and the application of this technology is not mature enough , It is very difficult to operate the VR system proficiently, and the accuracy and flexibility are relatively poor. At the same time, it is very tiring to hold the arm in the camera's field of view for a long time, and the user experience is poor.

6) When playing a game in a virtual scene, if there are other operation needs to bring up the menu, such as returning to the main interface to see other resources, you need to return after the current game is over. Even some players can't quickly and intuitively find a way to exit when playing games, and can't directly call up the main interface with one key.

Current VR interactive input methods generally have the problems of low sensitivity, low accuracy, poor response and inconvenient operation.

Fig. 15 is a scene diagram of a virtual reality interaction provided by related technologies. In VR scenarios, users often need to input some information to achieve VR interaction. As shown in Figure 15, the application scenario includes a helmet 11 and a peripheral device 12; among them, the user wears the helmet 11, and the helmet 11 can render a VR interface in the virtual scene, and the user can display the VR interface in the virtual scene through the peripheral device 12 Interact with the VR interface. For example, enter the account number and password; or adjust the volume and screen size while watching audio and video; or during the game, the peripheral device is equivalent to a prop in the game in the VR interface, such as swords, guns, etc. , The user inputs information to the VR game interface through the peripheral device, thereby controlling the props in the VR game interface.

In other scenes, users can also interact with VR through VR’s built-in touchpad, head motion hovering, voice recognition, and binocular recognition gestures, but these VR input methods generally suffer from inconvenience and sensitivity. Good, low accuracy, poor response. In particular, it is inconvenient to input passwords when making payments, entering text, or logging in to an account.

In response to the above-mentioned problems, the embodiments of the present application provide a virtual reality control device. There is a preset mapping relationship between the position points on the control device and the position points on the VR interface rendered by the VR helmet. The user can control For touch operation on the device, the user’s touch operation has position information. According to the touch position information and the preset mapping relationship, the user’s operation position in the VR interface can be determined. For the user, the user can be like It is very convenient to operate as usual on a computer screen or a mobile phone screen.

FIG. 16 is a control logic diagram of a virtual reality control device provided by an embodiment of the application. The embodiments of the present application provide a virtual reality control device in response to the above technical problems of related technologies. As shown in FIG. 16, the virtual reality control device includes: a touch interface 21-1, a controller 22-1, and a communicator 23 -1;

Among them, the touch interface 21-1 is used to receive user touch operation information.

The controller 22-1, connected to the touch interface 21-1, is used to determine the user's touch operation and operation position information on the touch interface based on the touch operation information.

The communicator 23-1, connected to the controller 22-1, is used to send the touch operation and operating position information to the processor of the helmet, so that the processor of the helmet is displayed on the VR interface rendered by the control device and the processor of the helmet The preset mapping relationship of the position points and the operation position information determine the corresponding position in the VR interface, and perform the operation behavior corresponding to the touch operation at the corresponding position. The communication mode of the communicator may be wired communication or wireless communication, which is not specifically limited in this embodiment.

In this embodiment, it can be understood that a touch interface is provided on the control device, and the touch interface has multiple touch position points, and the VR interface rendered by the helmet processor also has multiple VR position points. The preset mapping relationship between the position points on the touch interface and the position points on the VR interface can map the position points on the touch interface to the VR interface, and according to the user's operation behavior on the touch interface, the VR helmet Perform the corresponding operation at the corresponding position in the VR interface. For example, as shown in Figure 17, the A position point on the touch interface 21-1 corresponds to the B position point on the VR interface. If the user selects the A position point, then the B position point on the VR interface is also The selected operation will be performed. For another example, as shown in Figure 18, the user moves from the A1 point on the touch interface 21-1 along the track a1 (shown by the arc in the touch interface in Figure 18) to the A2 point, if the VR interface The B1 position point on the above corresponds to the A1 position point, the B2 position point corresponds to the A2 position point, and the trajectory v1 (as shown by the arc in the VR interface in Figure 18) corresponds to a1, then the VR interface will also execute from the B1 position point The operation of moving along the trajectory v1 to the position B2.

In some scenarios, the embodiment of the present application may also render the control device in a virtual reality scene. In this case, since the posture information of the control device may change at any time, in order to realize the posture tracking of the control device, an inertial measurement unit 24-1 may be added to the control device on the basis of the foregoing embodiment. As shown in Figure 19, the inertial measurement unit 24-1 is connected to the controller 22-1. The inertial measurement unit 24-1 is used to measure the angular velocity information of the control device and send it to the controller; the controller 22-1 is also used to The posture information of the control device is calculated based on the angular velocity information; the posture information is sent to the processor of the helmet, so that the processor of the helmet adjusts the posture information of the rendered VR interface based on the posture information.

In some scenarios, the position information of the control device may also change at any time. In order to track the position of the control device, the inertial measurement unit 24-1 can also be used to measure the acceleration information of the control device and send it to the controller 22. -1; and make the controller 22-1 calculate the position information of the control device based on the acceleration information, and then send the position information to the processor of the helmet, so that the processor of the helmet adjusts the position information of the rendered VR interface based on the position information.

Optionally, the inertial measurement unit 24-1 can also collect acceleration information and angular velocity information of the control device at the same time, and send them to the controller 22-1; and make the controller 22-1 calculate the position information of the control device based on the acceleration information, And calculate the attitude information of the control device based on angular velocity information; then send the position information and attitude information to the processor of the helmet together, so that the processor of the helmet adjusts the position information of the rendered VR interface based on the position information, and adjusts based on the attitude information The pose of the rendered VR interface.

For the inertial measurement unit 24-1, the attitude angle of the control device can be measured, and the attitude angle includes a roll angle (roll), a pitch angle (pitch), and a yaw angle (yaw). Among them, as shown in Figure 20, taking an airplane as an example, taking a point on the airplane as the origin, and taking the length of the airplane's fuselage as the Y axis, the horizontal plane perpendicular to the Y axis as the X axis, and the gravity direction as the Z axis. To establish an XYZ coordinate system, the pitch angle refers to the angle produced by rotating around the X axis, yaw refers to the angle produced by rotating around the Y axis, and roll refers to the angle produced by rotating around the Z axis. The inertial measurement unit includes a gyroscope, an accelerometer, and a magnetometer; among them, the gyroscope is used to measure the angular velocity, and the attitude angle can be obtained by integrating the angular velocity. However, due to errors in the integration process, as time increases, errors will also accumulate, eventually leading to attitude angle deviations. Therefore, the acceleration and gravity information of the control device can also be measured by the accelerometer. Therefore, the acceleration information can be used to correct the attitude angle deviation related to the direction of gravity, that is, the acceleration information can be used to correct the roll and pitch angles. The angle deviation. The measured data of the magnetometer can be calculated to obtain the yaw angle (yaw), and then the posture information can be corrected.

The control device of the embodiment of the present application can provide multiple control modes, and in different control modes, users can have different interactive experiences.

In an optional implementation manner, the control device corresponds to the first control mode; the control device may also be used to obtain the user’s status on the touch interface 21-1 when it is detected that the user selects the first control mode. And send it to the processor of the helmet so that the processor of the helmet determines the correspondence of the VR interface based on the touch position information and the preset mapping relationship between the position points of the VR interface rendered by the control device and the processor of the helmet Position, and move the cursor on the VR interface rendered in the virtual scene to the corresponding position. In this embodiment, the processor of the helmet renders the VR interface in the virtual scene, and the user can touch on the touch interface, just like a finger touching on the screen of a mobile phone, and the user’s touch action on the touch interface will generate a touch. Location information is used to indicate the user's touch position point on the touch interface. According to the touch position point and the preset mapping relationship between the position point on the touch interface of the touch interface and the position point of the VR interface, The corresponding location point of the VR interface can be determined.

As shown in FIG. 21, there are multiple touch position points on the touch interface (as shown by the dashed box in the figure). As shown in Figure 22, when the user touches multiple touch position points, touch operation information can be formed and sent to the helmet. The helmet can determine the VR master based on the touch operation information and the preset mapping relationship. The corresponding position on the interface.

On the basis of the above-mentioned embodiment, the user can also perform a confirmation operation after the touch operation. In this case, the control device can also be used to obtain the user's confirmation button information on the touch interface, and send it to the processor of the helmet, so that the processor of the helmet performs a confirmation operation on the current object on the VR interface. Optionally, the confirmation operation may be a click operation. For example, if the user touches on the touch interface and stays at a certain point to perform a click operation, it can be regarded as the user inputting confirmation button information on the touch interface.

In the first control mode, the processor of the helmet can render the VR interface to be displayed in a virtual scene. In this case, there is a preset proportional relationship between the touch interface and the VR interface. Wherein, the preset ratio relationship may be 1:1, may be greater than 1:1, and may also be less than 1:1, which is not specifically limited in this embodiment. Regardless of the preset ratio relationship, it can be mapped to the corresponding position of the VR interface according to the coordinates of the user's touch position on the control device and the preset ratio relationship.

The first control mode is for the user, the control device is like a mouse. The difference from the mouse is that the user moves the position of the VR interface by touching the control device, which can be understood as The touch movement on the control device is equivalent to using a mouse to move.

In another optional implementation manner, the control device corresponds to a second control mode; the control device can also be used to obtain the user's touch on the touch interface when it is detected that the user selects the second control mode. The control operation is sent to the processor of the helmet, so that the processor of the helmet performs corresponding operations on the corresponding position based on the user's touch operation.

In this embodiment, as shown in FIG. 23, the VR interface rendered by the processor of the helmet is located on the touch interface. When the user touches on the touch interface, it is like touching on the screen of a mobile phone. For the user, it is equivalent to directly operating on the VR interface. The difference from the first control mode is that the user can directly perform a click operation at any touch position, thereby realizing a confirmation operation. For other users wearing VR helmets, the VR interface rendered on the control device is invisible, and is only visible to the user wearing the VR helmet. When the user enters the account and password, it will be very safe.

In the second control mode, in order to enable the user to achieve a WYSIWYG experience effect, the touch interface and the VR interface can be in a preset ratio of 1:1.

In the second control mode, since the VR interface is rendered on the touch interface, in order to ensure that the VR interface can adapt to changes in the pose of the touch interface, the measurement based on the inertial measurement unit can be obtained in real time. The position information and posture information obtained from the data are adapted to adjust the position information and posture information of the VR interface based on the position information and posture information of the control device, so that the VR interface can change correspondingly with the change of the position and posture of the touch interface.

In order to further enhance the user experience, you can also design the first-point touch operation in the above two control modes, that is to say, when the user's touch operation is detected, the corresponding cursor, such as an arrow and a circle, will be displayed on the VR interface. , Mobile operations, etc., for users, it has a good visualization effect.

On the basis of the above two control modes, a mode selection function can also be provided to the user. For example, the user can freely choose whether to use the first control mode or the second control mode. In an optional implementation manner, a mode button may be set on the control device, and the user selects to use the first control mode or the second control mode by operating the mode button.

In the implementation manner in which the user selects the first control mode or the second control mode by operating the mode button, there may be the following optional implementation manners:

In the first optional implementation manner, the first mode button and the second mode button can be respectively set on the control device. When the user presses the first mode button, it means that the user chooses to use the first control mode; when the user presses Press the second mode button, it means that the user chooses to use the second control mode.

In a second optional implementation manner, a mode button can be provided on the control device, and the user can choose to use the first control mode or the second control mode by performing different operations on the mode button. Exemplarily, it is determined whether the user chooses to use the first control mode or the second control mode by setting the pressing time of the mode button. For example, when the user long presses the mode button, it means that the user chooses to use the first control mode, and when the user short presses the mode button, it means that the user chooses to use the second control mode. Among them, long press and short press are relative terms, that is, the time of short press is shorter than long press. For example, if the user presses the mode button and then releases it, it means that the user selects the first control mode. If the user presses the mode button for more than 5 seconds, it means that the user selects the second control mode.

In a third optional implementation manner, voice control can also be used to select the first control mode or the second control mode. For example, the user performs voice control by issuing a voice command "enter the first control mode" or "enter the second control mode".

In the fourth optional implementation manner, gesture control can also be used to select the first control mode or the second control mode. For example, a pulling gesture from bottom to top is to open the first control mode; a pulling gesture from left to right is to open the second control mode. Of course, other gestures can also be set to select the first control mode or the second control mode. This embodiment will not repeat them one by one again.

On the basis of the foregoing embodiment, in order to provide input feedback to the user, so that the user has a better sense of touch, a motor can also be provided in the control device. In this embodiment, the controller is also used to send a vibration instruction to the motor when the user's touch operation information is received; the motor is connected to the controller and is used to vibrate based on the vibration instruction. In this embodiment, it can be understood that when the user has an input operation on the touch interface, the controller gives feedback to the user's input, similar to the way of vibrating in the case of a tap operation on the mobile phone screen. In this way, it can provide users with a good sense of touch in a virtual scene.

In the above two control modes, the touch interface can adopt capacitive touch or infrared touch. Among them, the capacitive touch can refer to the existing capacitive touch technology, which will not be repeated here. For infrared touch, as shown in FIG. 24, the infrared light-emitting diodes and photodiodes arranged alternately are arranged on one side of the control device, and the control device also includes a control unit. The area shown by the square or rectangular dashed box in the figure is a touchable area. Among them, the light-emitting diode is used to emit infrared light. When a touch object is touched in the touchable area, such as a finger, the photodiode will receive the reflected light of the touch object to form an optical network. According to the emission and reception of infrared light , And the algorithm preset in the control unit performs calculations to determine the location of the touch point.

As shown in Figure 25, the control unit will continuously scan the infrared emitting tube and infrared receiving tube. When the human hand touches the touchable area, when the infrared light emitted by the infrared light-emitting diode meets the human hand, a part of the light will be reflected. A certain angle of photodiode is used to receive the infrared light reflected by the human hand, and then the control unit analyzes and calculates the position of the touch point according to the changes in the signal of the transmitting part and the receiving part.

In the embodiment of the present application, the processor of the helmet can make a synchronous response corresponding to the operation of the contact on the touch interface by the user. For example, when a user chooses to watch an immersive video on the main interface of VR, he controls the device to adjust the volume and the speed of the progress. When playing the game, call the projection interface to select resources and so on. For users, it is just like inputting information on the touch screen of a mobile phone, which conforms to the user's natural habits and can directly use the application without training.

On the basis of the two control modes provided by the foregoing embodiments, the embodiments of the present application may further include: detecting whether to exit the control mode, and if it is detected that the user wants to exit the control mode, long press the mode button again to exit the control mode. Or in the control mode, long time no touch operation can automatically exit the control mode.

The foregoing embodiment introduces a control device used for VR interaction in a virtual reality scene. The following will introduce a helmet used for VR interaction in a virtual reality scene.

Fig. 26 is a schematic diagram of the control logic of a helmet provided by an embodiment of the present application.

As shown in FIG. 26, the helmet provided by the embodiment of the present application includes: a processor 121-1 connected to a control device for rendering the VR interface, and when receiving the user's touch operation and operation position information on the control device In this case, according to the preset mapping relationship between the control device and the position point on the VR interface, and the received operation position information on the control device, determine the corresponding position in the VR interface, and perform the corresponding touch operation behavior at the corresponding position Operation behavior.

On the basis of the foregoing embodiment, the processor 121-1 of the embodiment of the present application is also used to obtain the position information and posture information of the control device, and based on the position information and posture information, determine the projection position and the projection position of the VR main interface to be displayed. Posture information, and based on the projection position and posture information of the VR main interface to be displayed, the posture information determined by the VR main interface to be displayed is projected to the corresponding position.

In the embodiments of the present application, for the pose tracking of the control device, at least two implementation manners as follows can be adopted:

In an alternative implementation manner, please continue to refer to FIG. 26. The helmet of the embodiment of the present application further includes: a camera 122-1 for collecting images including control equipment; a processor 121-1 connected to the camera 122- 1. It is used to determine the location information of the control device based on the image. In this embodiment, the location information of the control device refers to the location information of the control device in the world coordinate system. The processor determines the location information of the control device based on the image collected by the camera. Specifically, the image processing method is adopted. First, it is determined that the control device is located The position information in the image coordinate system is then converted to the world coordinate system based on the conversion relationship between the image coordinate system and the world coordinate system, so as to obtain the position information of the control device. Regarding how to determine the conversion relationship between the image coordinate system and the world coordinate system, reference may be made to the introduction of related technologies, which will not be repeated in this embodiment.

In another optional implementation manner, the processor 121-1 is further configured to obtain the position information and attitude information of the control device from the control device; the position information and attitude information of the control device are based on the acceleration information and the attitude information of the control device, respectively. The angular velocity information is calculated. In this embodiment, the acceleration information and angular velocity information measured by the inertial measurement unit of the control device are respectively integrated to obtain position information and posture information, and then sent to the processor of the helmet through the communicator of the control device.

Corresponding to the first control mode of the control device, the processor 121-1 is also used to render the VR interface in the virtual scene, and in the case of receiving the user's touch position information on the control device, based on the touch position information, and The preset mapping relationship between the control device and the position point of the VR interface rendered by the processor 121-1 determines the corresponding position of the VR interface, and moves the cursor on the VR interface rendered in the virtual scene to the corresponding position.

On the basis of the first control mode, the processor 121-1 is further configured to perform a confirmation operation on the current object on the VR interface in the case of receiving the confirmation key information of the user on the control device.

Corresponding to the second control mode of the control device, the processor 121-1 is also used to render the VR interface on the touch interface, and in the case of receiving the user’s touch operation on the touch interface, according to the user’s Touch operation, perform corresponding operations at the corresponding position on the VR interface.

For the specific implementations of the first control mode and the second control mode, please refer to the introduction of the control device part, which will not be repeated here. In addition, the introduction of the function of the helmet by the control device part can also be applied to the helmet of this embodiment, and this embodiment will not be repeated here.

In another embodiment of the present application, a virtual reality interactive system may also be provided. As shown in FIG. 27, the interactive system includes a virtual reality control device 131 and a helmet 132, wherein the control device 131 For the helmet 132, please refer to the introduction of the foregoing embodiment, which will not be repeated here.

FIG. 28 is a flowchart of a virtual reality interaction method provided by an embodiment of the application. On the basis of the foregoing embodiment, as shown in FIG. 28, the virtual reality interaction method provided by this embodiment specifically includes the following steps:

Step 1401: Receive touch operation information of the user.

The execution subject of this embodiment may be the control device in the foregoing embodiment. Among them, the user's touch operation information refers to the touch operation information on the control device.

Step 1402, based on the touch operation information, determine the user's touch operation and the operation position information on the control device.

Among them, touch operation refers to touch actions, such as operation behaviors such as moving, sliding, touching, and clicking, and operation position information refers to position information of the touch action on the control device.

Step 1403: Send the touch operation and operation position information to the processor of the helmet, so that the processor of the helmet determines the correspondence in the VR interface according to the preset mapping relationship between the control device and the position point on the VR interface, and the operation position information Position, and perform the operation behavior corresponding to the touch operation behavior at the corresponding position.

Optionally, the method of the embodiment of the present application further includes: measuring the angular velocity information of the control device; calculating the attitude information of the control device based on the angular velocity information; sending the attitude information to the processor of the helmet, so that the processor of the helmet is based on the attitude information Adjust the posture information of the rendered VR interface.

Optionally, the method of the embodiment of the present application further includes: measuring the acceleration information of the control device; calculating the position information of the control device based on the acceleration information; sending the position information to the processor of the helmet, so that the processor of the helmet is based on the position information Adjust the position information of the rendered VR interface.

Optionally, the control device corresponds to a first control mode; the method in this embodiment of the present application further includes: in a case where it is detected that the user selects the first control mode, acquiring the user's touch position information on the touch interface, and sending it to The processor of the helmet, so that the processor of the helmet can determine the corresponding position of the VR interface based on the touch position information and the preset mapping relationship between the position points of the VR interface rendered by the control device and the processor of the helmet, and set it in the virtual scene The cursor on the rendered VR interface moves to the corresponding position.

Optionally, the method of the embodiment of the present application further includes: in the case of detecting that the user selects the first control mode, acquiring the user's confirmation button information on the touch interface, and sending it to the processor of the helmet, so that the helmet The processor performs a confirmation operation on the current object on the VR interface.

Optionally, in the first control mode, the touch interface and the VR interface are in a preset proportional relationship.

Optionally, the control device corresponds to a second control mode; the method in this embodiment of the present application further includes: in the case of detecting that the user selects the second control mode, acquiring the user's touch operation on the touch interface, and sending it to The processor of the helmet enables the processor of the helmet to perform corresponding operations on the corresponding position based on the user's touch operation.

Optionally, in the second control mode, the touch interface and the VR interface are in a preset ratio relationship, and the preset ratio relationship is 1:1.

Optionally, the control device corresponds to multiple control modes; the method in this embodiment of the present application further includes: receiving user selection information for multiple control modes, and enabling the corresponding control mode based on the selection information; the selection information includes at least one of the following One: mode button selection, voice command selection and gesture selection.

Optionally, the method of the embodiment of the present application further includes: in the case of receiving the user's touch operation information, sending a vibration instruction to the motor, and controlling the motor to vibrate based on the vibration instruction.

The virtual reality interaction method of the embodiment shown in FIG. 28 can be used to implement the technical solutions of the foregoing control device embodiment, and its implementation principles and technical effects are similar, and will not be repeated here.

FIG. 29 is a flowchart of a virtual reality interaction method provided by another embodiment of this application. On the basis of the foregoing embodiment, as shown in FIG. 29, the virtual reality interaction method provided by this embodiment specifically includes the following steps:

Step 1501: Render the VR interface.

The execution subject of this embodiment may be the helmet of the above embodiment. Among them, the VR interface is rendered by the helmet.

Step 1502, in the case of receiving the user's touch operation and operation position information on the control device, according to the preset mapping relationship between the control device and the position point on the VR interface, and the received operation position information on the control device, Determine the corresponding position in the VR interface.

Step 1503: Perform an operation behavior corresponding to the touch operation behavior at the corresponding position.

For the specific implementation process of step 1502 and step 1503, reference may be made to the introduction of the foregoing embodiment, which will not be repeated here.

Optionally, the method of the embodiment of the present application further includes: acquiring position information and posture information of the control device; determining the projection position and posture information of the VR main interface to be displayed based on the position information and posture information; and based on the VR main interface to be displayed The projection position and posture information of the interface, the posture information determined by the VR main interface to be displayed is projected to the corresponding position.

Optionally, the method of the embodiment of the present application further includes: collecting an image including the control device; and determining the location information of the control device based on the image.

Optionally, the method of the embodiment of the present application further includes: acquiring position information and attitude information of the control device from the control device; the position information and attitude information of the control device are calculated based on the acceleration information and angular velocity information of the control device, respectively.

Optionally, the method of the embodiment of the present application further includes: rendering the VR interface in the virtual scene, and in the case of receiving the user's touch position information on the control device, based on the touch position information, and the processing of the control device and the helmet The preset mapping relationship of the position points of the VR interface rendered by the monitor is determined, the corresponding position of the VR interface is determined, and the cursor on the VR interface rendered in the virtual scene is moved to the corresponding position.

Optionally, the method of the embodiment of the present application further includes: in the case of receiving the confirmation key information of the user on the control device, performing a confirmation operation on the current object on the VR interface.

Optionally, the method of the embodiment of the present application further includes: rendering the VR interface on the touch interface, and in the case of receiving the user's touch operation on the touch interface, according to the user's touch operation, in the VR interface Perform the corresponding operation on the corresponding position.

The virtual reality interaction method of the embodiment shown in FIG. 29 can be used to implement the technical solution of the foregoing helmet embodiment, and its implementation principles and technical effects are similar, and will not be repeated here.

FIG. 30 is a schematic structural diagram of a virtual reality control device provided by an embodiment of the application. A virtual reality control device provided by an embodiment of the present application can execute the processing flow provided in an embodiment of a virtual reality control method as shown in FIG. 28. As shown in FIG. 30, a virtual reality control device 160 includes : A memory 161, a processor 162, a computer program, and a communication interface 163; wherein the computer program is stored in the memory 161 and is configured to be executed by the processor 162 to execute the processing provided by the embodiment of the virtual reality control method shown in FIG. 28 Process.

The virtual reality control device of the embodiment shown in FIG. 30 can be used to implement the technical solutions of the foregoing method embodiments, and its implementation principles and technical effects are similar, and will not be repeated here.

FIG. 31 is a schematic structural diagram of a virtual reality control device provided by an embodiment of this application. A virtual reality control device provided by an embodiment of the present application can execute the processing flow provided in an embodiment of a virtual reality control method as shown in FIG. 29. As shown in FIG. 31, a virtual reality control device 170 includes : A memory 171, a processor 172, a computer program, and a communication interface 173; wherein the computer program is stored in the memory 171 and is configured to be executed by the processor 172 to perform the processing provided by the embodiment of the virtual reality control method shown in FIG. 29 Process.

The virtual reality control device of the embodiment shown in FIG. 31 can be used to execute the technical solutions of the foregoing method embodiments, and its implementation principles and technical effects are similar, and will not be repeated here.

In addition, an embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the virtual reality interaction method of the embodiment shown in FIG. 28.

In addition, an embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the virtual reality interaction method of the embodiment shown in FIG. 29.

In the embodiments of the present application, the above-mentioned embodiments can refer to each other and learn from each other, and the same or similar steps and nouns will not be repeated one by one.

It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present application can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present application can be achieved, this is not limited herein.

The foregoing specific implementations do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principle of this application shall be included in the protection scope of this application.

Claims

A controller photosphere tracking method based on virtual reality, characterized in that, the method includes:

Determine, according to the first posture information of the previous location point of the photosphere, the second posture information of the next location point adjacent to the previous location point;

Determine the second location information of the next location point according to the first location information, the first posture information, and the second posture information of the previous location point;

According to the second position information, the current display position of the virtual target corresponding to the controller is generated and output.
The method according to claim 1, wherein the second position of the latter position point is determined according to the first position information, the first posture information and the second posture information of the previous position point. Location information, including:

According to the first posture information, the first predicted position when the photosphere is located at the previous position point is determined, wherein the first predicted position indicates that when the photosphere is located at the previous position point, relative The position of the initial position point;

According to the second posture information, a second predicted position when the photosphere is located at the latter position point is determined, wherein the second predicted position indicates that when the photosphere is located at the latter position point, relative The position of the initial position point;

According to the second predicted position and the first predicted position, the movement displacement of the photosphere is determined, wherein the movement displacement represents the movement of the photosphere from the previous position point to the latter position point Displacement

The second position information is determined according to the movement displacement and the first position information.
The method according to claim 2, wherein, according to the first posture information, determining the first predicted position when the photosphere is located at the previous position point comprises:

Determining the first predicted position according to the first posture information and a preset bone joint model, wherein the bone joint model is used to indicate the movement relationship of the human joints;

According to the second posture information, determining the second predicted position when the photosphere is located at the latter position point includes:

The second predicted position is determined according to the second posture information and the bone joint model.
The method according to claim 3, wherein the bone joint model includes a preset moving radius; and determining the first predicted position according to the first posture information and the preset bone joint model includes :

The first predicted position is determined according to the first posture information, the movement radius, and the preset first movement time, where the first movement time is the movement of the photosphere from the initial position point to The time required for the previous location point;

The determining the second predicted position according to the second posture information and the bone joint model includes:

The second predicted position is determined according to the second posture information, the movement radius, and the preset second movement time, where the second movement time is the movement of the photosphere from the initial position point to The time required for the latter location point.
The method according to claim 1, wherein, according to the first posture information of the previous position point of the photosphere, determining the second posture information of the next position point adjacent to the previous position point comprises:

Obtain the posture data detected by the inertial measurement unit;

The second posture information is determined according to the first posture information, the posture data, and the preset movement time, wherein the movement time is the movement of the light ball from the previous position to the back The time required for a location.
The method according to claim 5, wherein determining the second posture information according to the first posture information, the posture data, and a preset movement time comprises:

Determine a movement angle according to the posture data and the movement time;

The second posture information is determined according to the movement angle and the first posture information.
The method according to claim 5, wherein the attitude data is any one of the following: rotation angular velocity, gravitational acceleration, yaw angle, and pitch angle.
The method according to any one of claims 1-7, characterized in that, according to the first posture information of the previous position point of the photosphere, the second position point of the next position point adjacent to the previous position point is determined Before the second posture information, the method further includes:

Acquiring the first position information and the first posture information of the previous position point of the photosphere;

Wherein, obtaining the first position information of the previous position point of the photosphere includes:

Acquiring an image, where the image is an image collected by a collecting unit when the photosphere is located at the previous position point;

According to the image, the position of the photosphere in the image is determined to obtain the first position information.
A virtual reality device, which includes:

A display screen, which is used to display images;

A processor, the processor is configured to:

Determining, according to the first posture information of the previous position point of the photosphere on the controller, the second posture information of the next position point adjacent to the previous position point;

Determine the second location information of the next location point according to the first location information of the previous location point, the first posture information, and the second posture information;

According to the second location information, the location of the controller is determined, and then the screen is displayed.
The virtual reality device according to claim 9, wherein:

The processor is configured to:

Receive the posture data sent by the controller, and determine the second posture information according to the first posture information and the posture data.