CN117893567A - Tracking method, device and storage medium - Google Patents
Tracking method, device and storage medium Download PDFInfo
- Publication number
- CN117893567A CN117893567A CN202311749451.0A CN202311749451A CN117893567A CN 117893567 A CN117893567 A CN 117893567A CN 202311749451 A CN202311749451 A CN 202311749451A CN 117893567 A CN117893567 A CN 117893567A
- Authority
- CN
- China
- Prior art keywords
- mobile terminal
- detection frame
- tracking
- data
- mounted display
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000001514 detection method Methods 0.000 claims abstract description 204
- 230000000007 visual effect Effects 0.000 claims abstract description 28
- 230000001133 acceleration Effects 0.000 claims description 46
- 238000006073 displacement reaction Methods 0.000 claims description 44
- 239000011159 matrix material Substances 0.000 claims description 29
- 238000004590 computer program Methods 0.000 claims description 14
- 239000013598 vector Substances 0.000 claims description 14
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000005516 engineering process Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 230000003190 augmentative effect Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 4
- 239000011521 glass Substances 0.000 description 4
- 238000010295 mobile communication Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The application provides a tracking method, equipment and a storage medium, which belong to the field of visual tracking, and the method comprises the following steps: acquiring first sensing data output by a sensor of the mobile terminal at the current moment and second sensing data output by the sensor at the historical moment, wherein the current moment and the historical moment are separated by at least one preset time; determining motion data of the mobile terminal according to the first sensing data, the second sensing data and at least one preset time; acquiring a first detection frame of a mobile terminal, wherein the first detection frame is obtained by performing visual detection on the mobile terminal at a historical moment by using head-mounted display equipment; and tracking the mobile terminal according to the motion data and the first detection frame. According to the method and the device, the detection frame of the mobile terminal at the current moment can be accurately tracked according to the first detection frame and the motion data of the mobile terminal at the historical moment, and the mobile terminal is tracked and displayed based on the detection frame, so that the tracking efficiency and the tracking accuracy of the mobile terminal are greatly improved.
Description
Technical Field
The present disclosure relates to the field of visual tracking technologies, and in particular, to a tracking method, device, and storage medium.
Background
Augmented reality (Augmented Reality, AR) technology is one of hot spots in the field of computer application, and AR technology integrates real world information and virtual world information, for example, a head-mounted display device in the field of AR technology can track a mobile terminal used by a user in real time, enhance AR experience interacted with the mobile terminal, and when the head-mounted display device tracks the mobile terminal in real time, the current head-mounted display device may be affected by factors such as light, speed and shielding, so that the mobile terminal cannot be accurately tracked, and the use experience of the user is seriously affected.
Therefore, how to accurately track the mobile terminal is a problem to be solved.
Disclosure of Invention
The main purpose of the application is to provide a tracking method, a device and a storage medium, aiming at improving the accuracy of tracking a mobile terminal.
In a first aspect, the present application provides a tracking method applied to a head-mounted display device, where the head-mounted display device is communicatively connected to a mobile terminal, the method including the steps of:
acquiring first sensing data output by a sensor of the mobile terminal at the current moment and second sensing data output by the sensor at the historical moment, wherein the current moment and the historical moment are separated by at least one preset time;
Determining motion data of the mobile terminal according to the first sensing data, the second sensing data and the at least one preset time;
acquiring a first detection frame of the mobile terminal, wherein the first detection frame is obtained by the head-mounted display device performing visual detection on the mobile terminal at the historical moment;
and tracking the mobile terminal according to the motion data and the first detection frame.
In a second aspect, the present application further provides a tracking device, where the tracking device includes an acquisition module, a determination module, and a tracking module, where:
the acquisition module is used for acquiring first sensing data output by a sensor of the mobile terminal at the current moment and second sensing data output by the sensor of the mobile terminal at the historical moment, and the current moment and the historical moment are separated by at least one preset time;
the determining module is used for determining motion data of the mobile terminal according to the first sensing data, the second sensing data and the at least one preset time;
the acquisition module is further configured to acquire a first detection frame of the mobile terminal, where the first detection frame is obtained by performing visual detection on the mobile terminal at the historical moment by using a head-mounted display device;
And the tracking module is used for tracking the mobile terminal according to the motion data and the first detection frame.
In a third aspect, the present application also provides a head mounted display device comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program when executed by the processor implements the steps of the tracking method as described above.
In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the tracking method as described above.
The application provides a tracking method, equipment and a storage medium, wherein the tracking method, equipment and storage medium are applied to head-mounted display equipment, the head-mounted display equipment is in communication connection with a mobile terminal, first sensing data output by a sensor of the mobile terminal at the current moment and second sensing data output at the historical moment are obtained, and the current moment and the historical moment are separated by at least one preset time; determining motion data of the mobile terminal according to the first sensing data, the second sensing data and at least one preset time; acquiring a first detection frame of the mobile terminal, wherein the first detection frame is obtained by performing visual detection on the mobile terminal at a historical moment by using the head-mounted display equipment; and tracking the mobile terminal according to the motion data and the first detection frame. According to the method, the motion data in the motion process of the mobile terminal can be accurately obtained by calculating the first sensing data, the second sensing data and at least one preset time, the detection frame of the mobile terminal at the current moment can be accurately tracked according to the first detection frame and the motion data of the mobile terminal at the historical moment, the mobile terminal is tracked and displayed based on the detection frame, and the tracking efficiency and the tracking accuracy of the mobile terminal are greatly improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a tracking method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a scenario in which the tracking method according to the present embodiment is implemented;
FIG. 3 is a flow chart illustrating the sub-steps of step S102 in the embodiment of the present application;
FIG. 4 is a flow chart of the sub-steps of step S104 in the embodiment of the present application;
FIG. 5 is a flow chart illustrating a sub-step of sub-step S1044 in an embodiment of the present application;
FIG. 6 is a schematic block diagram of a tracking device provided in an embodiment of the present application;
FIG. 7 is a schematic block diagram of a tracking device sub-module in an embodiment of the present application;
fig. 8 is a schematic block diagram of a structure of a head-mounted display device according to an embodiment of the present application.
The realization, functional characteristics and advantages of the present application will be further described with reference to the embodiments, referring to the attached drawings.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The flow diagrams depicted in the figures are merely illustrative and not necessarily all of the elements and operations/steps are included or performed in the order described. For example, some operations/steps may be further divided, combined, or partially combined, so that the order of actual execution may be changed according to actual situations.
The embodiment of the application provides a tracking method, tracking equipment and a storage medium. The tracking method is applicable to a head-mounted display device, which may be augmented Reality (Augmented Reality, AR) glasses, AR helmets, mixed Reality (MR) glasses, MR helmets, and the like. For example, the head-mounted display device is AR glasses, the AR glasses are in communication connection with the mobile terminal, first sensing data output by a sensor of the mobile terminal at the current moment and second sensing data output by the sensor at the historical moment are obtained, and the current moment and the historical moment are separated by at least one preset time; determining motion data of the mobile terminal according to the first sensing data, the second sensing data and at least one preset time; acquiring a first detection frame of the mobile terminal, wherein the first detection frame is obtained by performing visual detection on the mobile terminal at a historical moment by using head-mounted display equipment; and tracking the mobile terminal according to the motion data and the first detection frame.
Some embodiments of the present application are described in detail below with reference to the accompanying drawings. The following embodiments and features of the embodiments may be combined with each other without conflict.
Referring to fig. 1, fig. 1 is a flow chart of a tracking method according to an embodiment of the present application.
As shown in fig. 1, the tracking method includes steps S101 to S104.
Step S101, first sensing data output by a sensor of the mobile terminal at the current moment and second sensing data output by the sensor at the historical moment are obtained, and the current moment and the historical moment are separated by at least one preset time.
The mobile terminal comprises electronic equipment such as a mobile phone, a tablet personal computer, a smart bracelet, a smart watch and the like, and is provided with a sensor for acquiring sensing data of the mobile terminal, wherein the sensing data at least comprises acceleration and angular velocity. For example, the sensor may be an inertial measurement unit (Inertial Measurement Unit, IMU), although the sensor may also be a sensor unit consisting of an acceleration sensor and an angular velocity sensor.
It should be noted that, the time interval between the historical time and the current time may be at least one preset time, and the preset time may be set according to practical situations, and the embodiment of the present invention is not limited to this, and the preset time may be, for example, 0.1 seconds. Under the condition that the preset time is 0.1 second and the historical time is spaced by one preset time, the current time is 10 hours, 00 minutes, 00 seconds and 100 milliseconds, and the historical time is 10 hours, 00 minutes, 00 seconds and 00 milliseconds; for another example, when the preset time is 0.1 seconds and the history time is three preset times apart, the current time is 10 hours 00 minutes 00 seconds 300 milliseconds, and the history time is 10 hours 00 minutes 00 seconds 00 milliseconds.
The manner of communication connection between the head-mounted display device and the mobile terminal may be set according to practical situations, and the embodiment of the present invention is not limited thereto, and for example, the manner of communication connection may be a fourth generation mobile communication technology (the 4th generation mobile communication technology,4G) connection, a fifth generation mobile communication technology (5 th-Generation Mobile Communication Technology, 5G) connection, a bluetooth connection, a wireless network communication technology (Wi-Fi) connection, and so on.
In one embodiment, first sensing data output by a sensor of a mobile terminal at a current moment is acquired, and second sensing data output by the sensor at a historical moment is acquired, wherein the sensing data comprises acceleration and angular velocity. By acquiring the sensing data output by the sensor of the mobile terminal, the first sensing data of the mobile terminal at the current moment and the second sensing data of the mobile terminal at the historical moment can be accurately obtained, and the tracking accuracy of the mobile terminal can be improved.
Step S102, determining motion data of the mobile terminal according to the first sensing data, the second sensing data and the at least one preset time.
The motion data is used for representing the change of the motion gesture of the mobile terminal, and comprises displacement data and rotation data. The first sensed data includes a first acceleration and a first angular velocity, and the second sensed data includes a second acceleration and a second angular velocity.
For example, as shown in fig. 2, in the case that the mobile terminal is a cellular phone, a three-axis rectangular coordinate system is established with a sensor of the cellular phone as an origin, and the three-axis rectangular coordinate system includes an x-axis, a y-axis and a z-axis. The coordinates of the x, y and z axes of the handset change during movement.
In one embodiment, as shown in fig. 3, step S102 includes sub-steps S1021 through S1022.
Substep S1021, determining displacement data of the mobile terminal according to the first acceleration, the second acceleration and the at least one preset time.
The first acceleration comprises a first acceleration in a first direction, a first acceleration in a second direction and a first acceleration in a third direction, and the second acceleration comprises a second acceleration in the first direction, a second acceleration in the second direction and a second acceleration in the third direction. The displacement data includes a displacement amount in the first direction, a displacement amount in the second direction, and a displacement amount in the third direction.
It should be noted that the first direction, the second direction and the third direction may be directions of an x axis, a y axis and a z axis of the space rectangular coordinate system according to practical situations, and the embodiment of the present invention is not limited in particular, for example, the first direction is an x axis direction, the second direction is a y axis direction and the first direction is a z axis direction.
In an embodiment, adding and averaging the first acceleration in the first direction and the second acceleration in the first direction to obtain an average value of the accelerations in the first direction; adding the first acceleration in the second direction and the second acceleration in the second direction, and taking an average value to obtain an average value of the accelerations in the second direction; and adding the first acceleration in the third direction and the second acceleration in the third direction, and taking the average value to obtain the average value of the accelerations in the third direction. The average value of the first acceleration and the second acceleration in each direction can be obtained, so that the acceleration in each direction in the moving process of the mobile terminal can be accurately obtained, and the accuracy of the moving detection of the mobile terminal is greatly improved.
In one embodiment, multiplying the average value of the acceleration in the first direction by at least one square of a preset time, and multiplying the average value of the acceleration by one half to obtain a displacement in the first direction; multiplying the average value of the acceleration in the second direction by at least one square of preset time and multiplying the average value of the acceleration by one half to obtain the displacement in the second direction; multiplying the average value of the acceleration in the third direction by at least one square of preset time and multiplying by one half to obtain the displacement in the third direction. And respectively carrying out displacement calculation on the acceleration average value in the first direction, the acceleration average value in the second direction and the acceleration average value in the third direction in an acceleration displacement calculation mode, so that displacement data of the mobile terminal can be accurately obtained.
Substep S1022, determining rotation data of the mobile terminal according to the first angular velocity, the second angular velocity, and the at least one preset time.
Wherein the first angular velocity comprises a first angular velocity about a first directional axis, a first angular velocity about a second directional axis, and a first angular velocity about a third directional axis; the second angular velocity includes a second angular velocity about the first direction axis, a second angular velocity about the second direction axis, and a second angular velocity about the third direction axis; the rotation data includes first rotation data, second rotation data, and third rotation data. The first rotation data, the second rotation data and the third rotation data are angular movement amounts of the mobile terminal around a coordinate axis, the first rotation data may be angular movement amounts around an x-axis, the second rotation data may be angular movement amounts around a y-axis, and the third rotation data may be angular movement amounts around a z-axis.
In one embodiment, subtracting the first angular velocity about the first direction axis from the second angular velocity about the first direction axis yields an angular velocity difference about the first direction axis; subtracting the first angular velocity around the second direction axis from the second angular velocity around the second direction axis to obtain an angular velocity difference around the second direction axis; subtracting the first angular velocity about the third direction axis from the second angular velocity about the third direction axis yields an angular velocity difference about the third direction axis.
It should be noted that, the radian conversion is performed on the angular velocity difference value around the first direction axis, the angular velocity difference value around the second direction axis, and the angular velocity difference value around the third direction axis, so as to improve the accuracy of calculating the motion data of the mobile terminal. Specifically, the angular velocity difference value is multiplied by pi/180 to complete the radian conversion.
In one embodiment, the angular velocity difference about the first directional axis is multiplied by at least one preset time to obtain first rotation data about the first directional axis; multiplying the angular velocity difference around the second direction axis by at least one preset time to obtain second rotation data around the second direction axis; and multiplying the angular velocity difference around the third direction axis by at least one preset time to obtain third rotation data around the third direction axis.
Step S103, a first detection frame of the mobile terminal is obtained, wherein the first detection frame is obtained by the head-mounted display device performing visual detection on the mobile terminal at the historical moment.
The first detection frame is obtained by performing visual detection on the mobile terminal at a historical moment by using a camera, and is used for determining a display area of the mobile terminal on a display device by using the head-mounted display device, the first detection frame comprises a coordinate set of the mobile terminal on the display device of the head-mounted display device, the coordinate system where the coordinate set is located can be established according to practical conditions, for example, a rectangular coordinate system is established by taking a central point of the display device as an origin, the coordinate set at least comprises coordinates of two vertexes of a diagonal line of the first detection frame, wherein the first vertex is an upper left corner of the first detection frame, and the first vertex is a lower right corner of the first detection frame.
In an embodiment, the head-mounted display device performs visual detection on the mobile terminal at a historical moment to obtain a detection frame of the mobile terminal, and takes the detection frame as a first detection frame. The first detection frame can be accurately obtained by performing visual detection on the mobile terminal through the camera of the head-mounted display device.
It should be noted that, the historical time is separated from the current time by at least one preset time, for example, a visual detection frame is generated every 0.1 seconds, and the historical time may be a time point corresponding to a frame previous to the current time or a time point corresponding to three frames previous to the current time.
In an embodiment, the target detection frame is obtained by tracking the mobile terminal by the head-mounted display device at the previous moment, so as to obtain the first detection frame of the mobile terminal at the previous moment. In other words, the embodiment can track the mobile terminal at the current moment through the target detection frame obtained by tracking at the previous moment so as to generate an updated target detection frame, thereby greatly improving the diversity and accuracy of mobile terminal tracking.
It should be noted that, the head-mounted display device at the previous moment tracks the mobile terminal to obtain a target detection frame, so as to obtain a first detection frame of the mobile terminal at the previous moment as follows: the head-mounted display device calculates a target detection frame based on the sensing data at the previous moment, and takes the calculated target detection frame as a first detection frame of the mobile terminal at the previous moment.
The first detection frame obtained by the head-mounted display device for carrying out visual detection on the mobile terminal at the historical moment is as follows: the head-mounted display device collects images of the mobile terminal through the camera device at historical moments and takes a detection frame generated based on the collected images as a first detection frame.
And step S104, tracking the mobile terminal according to the motion data and the first detection frame.
In one embodiment, as shown in fig. 4, step S104 includes sub-steps S1041 to S1044.
And step S1041, obtaining a distance value between the head-mounted display device and the mobile terminal and an image center coordinate when the head-mounted display device displays an image.
In an embodiment, a distance value of the head mounted display device from the mobile terminal is obtained in order to determine depth information.
It should be noted that, the manner of acquiring the distance value between the head-mounted display device and the mobile terminal may be selected according to practical situations, which is not limited in the embodiment of the present invention, for example, the distance value between the mobile terminal and the head-mounted display device is acquired through a sensor, or the arm length of the user is set to be the distance value between the mobile terminal and the head-mounted display device.
In an embodiment, the head-mounted display device performs image center coordinates when displaying pixels, and the image center coordinates are planar two-dimensional coordinates.
Step S1042, acquiring a first detection frame coordinate set of the first detection frame when the image is displayed in the head-mounted display device.
The first detection frame coordinate set is a coordinate set of the first detection frame when the image of the first detection frame is displayed in the head-mounted display device, and the coordinate set at least includes coordinates of two points of a diagonal line of the first detection frame, of course, the first detection frame coordinate set may include more coordinates, and the more the coordinates, the more the position of the first detection frame is, the more accurate the coordinates, in this embodiment, the upper left corner coordinate and the lower right corner coordinate of the first detection frame are taken as examples, but the embodiments in this application are not limited.
In an embodiment, a first coordinate of an upper left corner and a second coordinate of a lower right corner of the first detection frame when an image is displayed in the head-mounted display device are obtained, a first detection frame coordinate set is obtained, and the first detection frame can be accurately positioned by obtaining the first detection frame coordinate set, so that the tracking accuracy of the mobile terminal is greatly improved.
And step S1043, determining a first space coordinate set corresponding to the first detection frame according to the center coordinates of the image, the first detection frame coordinate set, the distance value and the focal length of the head-mounted display device.
The image center coordinate comprises a first direction image center coordinate value and a second direction image center coordinate value, wherein the first direction is the x-axis direction of the rectangular coordinate system, and the second direction is the y-axis direction of the rectangular coordinate system; or the first direction is the y-axis direction of the rectangular coordinate system, the second direction is the x-axis direction of the rectangular coordinate system, the first detection frame coordinate set comprises a first coordinate and a second coordinate, the first coordinate comprises a first direction first coordinate value and a second direction first coordinate value, the second coordinate comprises a first direction second coordinate value and a second direction second coordinate value, and the focal distance comprises a first direction focal distance and a second direction focal distance.
In one embodiment, the first coordinate value of the first spatial coordinate is determined according to the first direction first coordinate value, the first direction image center coordinate value, the first direction focal length and the distance value; determining a second coordinate value of the first space coordinate according to the first coordinate value in the second direction, the second image center coordinate value, the second direction focal length and the distance value; determining the distance value as a third coordinate value of the first spatial coordinate; and sequentially arranging and combining the first coordinate value of the first space coordinate, the second coordinate value of the first space coordinate and the third coordinate value of the first space coordinate to generate the first space coordinate.
For example, the first coordinate value of the first space coordinate is a, the second coordinate value of the first space coordinate is b, the third coordinate value of the first space coordinate is c, and the first coordinate value a, the second coordinate value b and the third coordinate value c of the first space coordinate are sequentially arranged and combined to generate the first space coordinate (a, b, c).
In an embodiment, the determining the first coordinate value of the first spatial coordinate according to the first coordinate value of the first direction, the central coordinate value of the first direction image, the first direction focal length and the distance value may be: subtracting the first direction image center coordinate value from the first direction first coordinate value to obtain a first direction coordinate difference; and multiplying the first direction coordinate difference value by the distance value and dividing the distance value by the first direction focal length to obtain a first coordinate value of the first space coordinate. By calculating the first-direction first coordinate value, the first-direction image center coordinate value, the first-direction focal length, and the distance value, the first coordinate value of the first spatial coordinate can be accurately obtained.
Exemplary, a first coordinate value formula of the preset space coordinate is obtained, wherein the first coordinate value formula of the preset space coordinate is that Wherein X is the first coordinate of the space coordinatesThe value x is the first coordinate value in the first direction, c x For the first direction image center coordinate value, d is the distance value and f x And calculating a first coordinate value of the first direction, a central coordinate value of the first direction image, a first direction focal length and a distance value based on a first coordinate value formula of the preset space coordinate to obtain the first coordinate value of the first space coordinate.
In an embodiment, the determining the second coordinate value of the first spatial coordinate according to the first coordinate value of the second direction, the central coordinate value of the second direction, the focal length of the second direction and the distance value may be: subtracting the central coordinate value of the second direction image from the second coordinate value of the first direction to obtain a second direction coordinate difference value; and multiplying the second direction coordinate difference value by the distance value and dividing the distance value by the second direction focal length to obtain a second coordinate value of the first space coordinate. The second coordinate value of the first space coordinate can be accurately obtained by calculating the first coordinate value of the second direction, the center coordinate value of the second direction image, the focal length of the second direction and the distance value.
Exemplary, a second coordinate value formula of the preset space coordinate is obtained, wherein the second coordinate value formula of the preset space coordinate is that Wherein Y is the second coordinate value of the space coordinate, Y is the first coordinate value of the second direction, c y For the central coordinate value of the second direction image, d is the distance value and f y And calculating a first coordinate value in the second direction, a central coordinate value of the second direction image, a focal length in the second direction and a distance value based on a second coordinate value formula of the preset space coordinate to obtain a second coordinate value of the first space coordinate.
And step S1044, tracking the mobile terminal according to the displacement data, the rotation data and the first space coordinate set to obtain a target detection frame of the mobile terminal at the current moment.
In one embodiment, as shown in fig. 5, the substep S1044 includes substeps S1044a to S1044b.
Substep S1044a, determining a second set of spatial coordinates after tracking the mobile terminal based on the displacement data, the rotation data and the first set of spatial coordinates.
Wherein the second spatial coordinates include at least a third coordinate of an upper left corner and a fourth coordinate of a lower right corner of the tracking frame.
In one embodiment, the rotation data is subjected to matrix construction to generate a rotation matrix, and vector conversion is performed on the displacement data to generate a displacement vector; multiplying the rotation matrix and each first space coordinate respectively, and adding the displacement vectors to generate a plurality of second space coordinates; and collecting each second space coordinate to generate the second space coordinate set. By calculating the displacement data, the rotation data, and the first spatial coordinates, respectively, the second spatial coordinate set can be accurately obtained.
In an embodiment, the rotation data is matrix-constructed, and the generation of the rotation matrix may be: the rotation matrix comprises a first rotation matrix, a second rotation matrix and a third rotation matrix; for the first rotation data theta x Performing matrix construction to generate a first rotation matrixFor the second rotation data theta y Performing matrix construction to generate a second rotation matrixFor the third rotation data theta z Performing matrix construction to generate a third rotation matrixAnd multiplying the first rotation matrix, the second rotation matrix and the third rotation matrix to obtain a rotation matrix.
For example, a preset second spatial coordinate formula is obtained, where m=r×n+t, M is a second spatial coordinate, R is a rotation matrix, N is a first spatial coordinate, and T is a displacement vector. And calculating the rotation matrix, the first space coordinate and the displacement vector based on the preset second space coordinate formula to obtain a second space coordinate.
And step S1044b, converting the second space coordinate set into a second detection frame coordinate set, and displaying the target detection frame of the mobile terminal at the current moment according to the second detection frame coordinate set.
In an embodiment, each spatial coordinate in the second spatial coordinate set is converted into a two-dimensional coordinate to obtain a second detection frame coordinate set, and a target detection frame of the mobile terminal at the current moment is displayed based on each coordinate position in the second detection frame coordinate set, so that the mobile terminal is accurately tracked.
In an embodiment, the method for converting each spatial coordinate in the second spatial coordinate set into a two-dimensional coordinate to obtain the second detection frame coordinate set may be: acquiring a first coordinate value formula of a preset two-dimensional coordinate and a second coordinate value formula of the preset two-dimensional coordinate, wherein the first coordinate value formula of the preset two-dimensional coordinate is as followsThe second coordinate value formula of the preset two-dimensional coordinate is +.>m is the first coordinate value of the second detection frame coordinate, n is the first coordinate value of the second detection frame coordinate, A is the first coordinate value of the second space coordinate, B is the second coordinate value of the second space coordinate, Z is the first coordinate value of the second space coordinate, c x For the central coordinate value, c, of the first direction image y Is the central coordinate value f of the second direction image x For a first direction focal length, f y And for the focal length in the second direction, calculating each space coordinate in the second space coordinate set based on the first coordinate value formula of the preset two-dimensional coordinate and the second coordinate value formula of the preset two-dimensional coordinate to obtain a second detection frame coordinate set.
In the embodiment of the invention, the first detection frame for visual detection of the mobile terminal can be tracked based on the historical moment to generate the target detection frame at the current moment, and the target detection frame at the current moment can be tracked based on the target detection frame obtained by tracking the mobile terminal at the previous moment, so that the flexibility and accuracy of tracking the mobile terminal are greatly improved.
In an embodiment, a second detection frame of the mobile terminal is obtained, wherein the second detection frame is obtained by performing visual detection on the mobile terminal at the current moment by using the head-mounted display device; determining the cross-over ratio between the second detection frame and the target detection frame obtained by tracking at the current moment; and when the cross-over ratio is smaller than the preset cross-over ratio, replacing the target detection frame obtained by tracking at the current moment with the second detection frame, namely taking the second detection frame as a new target detection frame, and tracking the mobile terminal through the new target detection frame. When the cross-over ratio is greater than or equal to the preset cross-over ratio, the original target detection frame can be continuously adopted to track the mobile terminal. The preset cross-over ratio may be set according to practical situations, which is not limited in the embodiment of the present invention, for example, the preset cross-over ratio may be set to 0.5. It can be understood that when the cross-over ratio is smaller than the preset cross-over ratio, the error of the target detection frame is indicated to be larger, so that the new target detection frame is required to track the mobile terminal. Therefore, the visual tracking result can be updated by comparing the target detection frame obtained by tracking with the second detection frame obtained by visual detection, and the efficiency and the accuracy of mobile terminal tracking are greatly improved.
In an embodiment, after tracking the mobile terminal, updating the first detection frame to a target detection frame obtained by tracking at the current moment; or updating the first detection frame into a detection frame obtained by the head-mounted display device for visually detecting the mobile terminal at the current moment. By correcting the first detection frame, the tracking of the mobile terminal can be conveniently performed based on the target detection frame at the current moment at the next moment, and the tracking efficiency of the mobile terminal is greatly improved.
The tracking method provided by the embodiment of the invention obtains the first sensing data output by the sensor of the mobile terminal at the current moment and the second sensing data output by the sensor of the mobile terminal at the historical moment, wherein the current moment and the historical moment are separated by at least one preset time; determining motion data of the mobile terminal according to the first sensing data, the second sensing data and at least one preset time; acquiring a first detection frame of the mobile terminal, wherein the first detection frame is obtained by performing visual detection on the mobile terminal at a historical moment by using the head-mounted display equipment; and tracking the mobile terminal according to the motion data and the first detection frame. According to the method, the motion data in the motion process of the mobile terminal can be accurately obtained by calculating the first sensing data, the second sensing data and at least one preset time, the detection frame of the mobile terminal at the current moment can be accurately tracked according to the first detection frame and the motion data of the mobile terminal at the historical moment, the mobile terminal is tracked based on the detection frame, and the tracking efficiency and accuracy of the mobile terminal are greatly improved.
Referring to fig. 6, fig. 6 is a schematic block diagram of a tracking device according to an embodiment of the present application.
As shown in fig. 6, the tracking device 200 includes an acquisition module 210, a determination module 220, and a tracking module 230, where:
the acquiring module 210 is configured to acquire first sensing data output by a sensor of the mobile terminal at a current time and second sensing data output by the sensor at a historical time, where the current time and the historical time are separated by at least one preset time;
the determining module 220 is configured to determine motion data of the mobile terminal according to the first sensing data, the second sensing data, and the at least one preset time;
the obtaining module 210 is further configured to obtain a first detection frame of the mobile terminal, where the first detection frame is obtained by performing visual detection on the mobile terminal at the historical moment by using a head-mounted display device;
the tracking module 230 is configured to track the mobile terminal according to the motion data and the first detection frame.
In an embodiment, the tracking device 200 is further configured to:
acquiring a second detection frame of the mobile terminal, wherein the second detection frame is obtained by the head-mounted display device performing visual detection on the mobile terminal at the current moment;
Determining the cross ratio between the second detection frame and the target detection frame obtained by tracking at the current moment;
and when the cross-over ratio is smaller than a preset cross-over ratio, replacing the target detection frame obtained by tracking at the current moment with the second detection frame.
In an embodiment, the tracking device 200 is further configured to:
updating the first detection frame into the current moment tracking to obtain a target detection frame;
or updating the first detection frame to a detection frame obtained by the head-mounted display device performing visual detection on the mobile terminal at the current moment.
In an embodiment, the determining module 220 is further configured to:
determining displacement data of the mobile terminal according to the first acceleration, the second acceleration and the at least one preset time;
and determining the rotation data of the mobile terminal according to the first angular speed, the second angular speed and the at least one preset time.
In one embodiment, as shown in fig. 7, the tracking module 230 includes an obtaining sub-module 231, a determining sub-module 232, and a generating module 233, where:
the acquiring sub-module 231 is configured to acquire a distance value between the head-mounted display device and the mobile terminal and an image center coordinate when the head-mounted display device displays an image;
The acquiring sub-module 231 is further configured to acquire a first detection frame coordinate set of the first detection frame when the image is displayed in the head-mounted display device;
the determining submodule 232 is configured to determine a first spatial coordinate set corresponding to the first detection frame according to the image center coordinate, the first detection frame coordinate set, the distance value and the focal length of the head-mounted display device;
the generating module 233 is configured to track the mobile terminal according to the displacement data, the rotation data, and the first set of spatial coordinates, so as to obtain a target detection frame of the mobile terminal at the current moment.
In an embodiment, the generating module 233 is further configured to:
determining a second space coordinate set after tracking the mobile terminal according to the displacement data, the rotation data and the first space coordinate set;
and converting the second space coordinate set into a second detection frame coordinate set, and displaying a target detection frame of the mobile terminal at the current moment according to the second detection frame coordinate set.
In an embodiment, the generating module 233 is further configured to:
constructing a matrix of the rotation data to generate a rotation matrix, and performing vector conversion on the displacement data to generate a displacement vector;
Multiplying the rotation matrix and each first space coordinate respectively, and adding the displacement vector to generate a plurality of second space coordinates;
and collecting the second space coordinates to generate a second space coordinate set.
It should be noted that, for convenience and brevity of description, a specific working process of the tracking device may refer to a corresponding process in the foregoing embodiment of the tracking method, which is not described herein again.
Referring to fig. 8, fig. 8 is a schematic block diagram of a structure of a head-mounted display device according to an embodiment of the present application.
As shown in fig. 8, the head mounted display device 300 includes a processor 302 and a memory 303 connected through a system bus 301, wherein the memory 303 may include a storage medium and an internal memory.
The storage medium may store a computer program. The computer program comprises program instructions that, when executed, cause a processor to perform any of a number of tracking methods.
The processor 302 is used to provide computing and control capabilities to support the operation of the overall head mounted display device 300.
The internal memory provides an environment for the execution of a computer program in a storage medium that, when executed by a processor, causes the processor to perform any of a number of tracking methods.
Those skilled in the art will appreciate that the structure shown in fig. 8 is merely a block diagram of a portion of the structure associated with the present application and is not limiting of the head mounted display device to which the present application is applied, and that a particular head mounted display device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
It should be appreciated that the processor 302 may be a central processing unit (Central Processing Unit, CPU), which may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Wherein in one embodiment, the processor 302 is configured to execute a computer program stored in a memory to implement the steps of:
acquiring first sensing data output by a sensor of the mobile terminal at the current moment and second sensing data output by the sensor at the historical moment, wherein the current moment and the historical moment are separated by at least one preset time;
Determining motion data of the mobile terminal according to the first sensing data, the second sensing data and the at least one preset time;
acquiring a first detection frame of the mobile terminal, wherein the first detection frame is obtained by the head-mounted display device performing visual detection on the mobile terminal at the historical moment;
and tracking the mobile terminal according to the motion data and the first detection frame.
In one embodiment, after implementing the tracking of the mobile terminal according to the motion data and the first detection box, the processor 302 is further configured to implement:
acquiring a second detection frame of the mobile terminal, wherein the second detection frame is obtained by the head-mounted display device performing visual detection on the mobile terminal at the current moment;
determining the cross ratio between the second detection frame and the target detection frame obtained by tracking at the current moment;
and when the cross-over ratio is smaller than a preset cross-over ratio, replacing the target detection frame obtained by tracking at the current moment with the second detection frame.
In one embodiment, after implementing the tracking of the mobile terminal according to the motion data and the first detection box, the processor 302 is further configured to implement:
Updating the first detection frame into a target detection frame obtained by tracking the current moment;
or updating the first detection frame to a detection frame obtained by the head-mounted display device performing visual detection on the mobile terminal at the current moment.
In one embodiment, the processor 302 is implementing that the first sensed data comprises a first acceleration and a first angular velocity, the second sensed data comprises a second acceleration and a second angular velocity, and the motion data comprises displacement data and rotation data; and when determining the motion data of the mobile terminal according to the first sensing data, the second sensing data and the at least one preset time, the method is used for realizing:
determining displacement data of the mobile terminal according to the first acceleration, the second acceleration and the at least one preset time;
and determining the rotation data of the mobile terminal according to the first angular speed, the second angular speed and the at least one preset time.
In one embodiment, when the processor 302 performs tracking on the mobile terminal according to the motion data and the first detection frame to obtain a target detection frame of the mobile terminal at the current time, the processor is configured to perform:
Acquiring a distance value between the head-mounted display device and the mobile terminal and an image center coordinate when the head-mounted display device performs pixel display;
acquiring a first detection frame coordinate set of the first detection frame when pixels in the head-mounted display device are displayed;
determining a first space coordinate set corresponding to the first detection frame according to the image center coordinate, the first detection frame coordinate set, the distance value and the focal length of the head-mounted display device;
and tracking the mobile terminal according to the displacement data, the rotation data and the first space coordinate set to obtain a target detection frame of the mobile terminal at the current moment.
In one embodiment, when the processor 302 performs tracking on the mobile terminal according to the displacement data, the rotation data and the first set of spatial coordinates to obtain the target detection frame of the mobile terminal at the current time, the processor is configured to perform:
determining a second space coordinate set after tracking the mobile terminal according to the displacement data, the rotation data and the first space coordinate set;
and converting the second space coordinate set into a second detection frame coordinate set, and displaying a target detection frame of the mobile terminal at the current moment according to the second detection frame coordinate set.
In one embodiment, the processor 302 includes a plurality of first spatial coordinates in implementing the first set of spatial coordinates; and when determining the second space coordinate set tracked by the mobile terminal according to the displacement data, the rotation data and the first space coordinate set, the method is used for realizing:
constructing a matrix of the rotation data to generate a rotation matrix, and performing vector conversion on the displacement data to generate a displacement vector;
multiplying the rotation matrix and each first space coordinate respectively, and adding the displacement vector to generate a plurality of second space coordinates;
and collecting the second space coordinates to generate a second space coordinate set.
It should be noted that, for convenience and brevity of description, the specific working process of the head-mounted display device described above may refer to the corresponding process in the foregoing tracking method embodiment, which is not described herein again.
Embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, where the computer program includes program instructions, where the method implemented when the program instructions are executed may refer to various embodiments of the tracking method of the present application.
The computer readable storage medium may be an internal storage unit of the head-mounted display device according to the foregoing embodiment, for example, a hard disk or a memory of the head-mounted display device. The computer readable storage medium may be nonvolatile or may be volatile. The computer readable storage medium may also be an external storage device of the head-mounted display device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, which are provided on the head-mounted display device.
It is to be understood that the terminology used in the description of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should also be understood that the term "and/or" as used in this specification refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments. While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention.
Claims (10)
1. A tracking method for use with a head mounted display device communicatively coupled to a mobile terminal, the method comprising:
acquiring first sensing data output by a sensor of the mobile terminal at the current moment and second sensing data output by the sensor at the historical moment, wherein the current moment and the historical moment are separated by at least one preset time;
determining motion data of the mobile terminal according to the first sensing data, the second sensing data and the at least one preset time;
acquiring a first detection frame of the mobile terminal, wherein the first detection frame is obtained by the head-mounted display device performing visual detection on the mobile terminal at the historical moment;
And tracking the mobile terminal according to the motion data and the first detection frame.
2. The tracking method of claim 1, wherein the tracking of the mobile terminal based on the motion data and the first detection frame further comprises:
acquiring a second detection frame of the mobile terminal, wherein the second detection frame is obtained by the head-mounted display device performing visual detection on the mobile terminal at the current moment;
determining the cross ratio between the second detection frame and the target detection frame obtained by tracking at the current moment;
and when the cross-over ratio is smaller than a preset cross-over ratio, replacing the target detection frame obtained by tracking at the current moment with the second detection frame.
3. The tracking method of claim 1, wherein the tracking of the mobile terminal based on the motion data and the first detection frame further comprises:
updating the first detection frame into the current moment tracking to obtain a target detection frame;
or updating the first detection frame to a detection frame obtained by the head-mounted display device performing visual detection on the mobile terminal at the current moment.
4. The tracking method of claim 1, wherein the first sensed data comprises a first acceleration and a first angular velocity, the second sensed data comprises a second acceleration and a second angular velocity, and the motion data comprises displacement data and rotation data; the determining the motion data of the mobile terminal according to the first sensing data, the second sensing data and the at least one preset time comprises the following steps:
determining displacement data of the mobile terminal according to the first acceleration, the second acceleration and the at least one preset time;
and determining the rotation data of the mobile terminal according to the first angular speed, the second angular speed and the at least one preset time.
5. The tracking method of claim 4, wherein tracking the mobile terminal based on the motion data and the first detection box comprises:
acquiring a distance value between the head-mounted display device and the mobile terminal and an image center coordinate when the head-mounted display device displays images;
acquiring a first detection frame coordinate set of the first detection frame when the image of the first detection frame is displayed in the head-mounted display device;
Determining a first space coordinate set corresponding to the first detection frame according to the image center coordinate, the first detection frame coordinate set, the distance value and the focal length of the head-mounted display device;
and tracking the mobile terminal according to the displacement data, the rotation data and the first space coordinate set to obtain a target detection frame of the mobile terminal at the current moment.
6. The tracking method according to claim 5, wherein tracking the mobile terminal according to the displacement data, the rotation data and the first set of spatial coordinates to obtain a target detection frame of the mobile terminal at the current time, includes:
determining a second space coordinate set after tracking the mobile terminal according to the displacement data, the rotation data and the first space coordinate set;
and converting the second space coordinate set into a second detection frame coordinate set, and displaying a target detection frame of the mobile terminal at the current moment according to the second detection frame coordinate set.
7. The tracking method of claim 6, wherein the first set of spatial coordinates comprises a plurality of first spatial coordinates; the determining a second space coordinate set tracked by the mobile terminal according to the displacement data, the rotation data and the first space coordinate set comprises the following steps:
Constructing a matrix of the rotation data to generate a rotation matrix, and performing vector conversion on the displacement data to generate a displacement vector;
multiplying the rotation matrix and each first space coordinate respectively, and adding the displacement vector to generate a plurality of second space coordinates;
and collecting the second space coordinates to generate a second space coordinate set.
8. A tracking device, characterized in that the tracking device comprises an acquisition module, a determination module and a tracking module, wherein:
the acquisition module is used for acquiring first sensing data output by a sensor of the mobile terminal at the current moment and second sensing data output by the sensor of the mobile terminal at the historical moment, and the current moment and the historical moment are separated by at least one preset time;
the determining module is used for determining motion data of the mobile terminal according to the first sensing data, the second sensing data and the at least one preset time;
the acquisition module is further configured to acquire a first detection frame of the mobile terminal, where the first detection frame is obtained by performing visual detection on the mobile terminal at the historical moment by using a head-mounted display device;
And the tracking module is used for tracking the mobile terminal according to the motion data and the first detection frame.
9. A head mounted display device comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program when executed by the processor implements the steps of the tracking method of any of claims 1 to 7.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program, wherein the computer program, when executed by a processor, implements the steps of the tracking method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311749451.0A CN117893567A (en) | 2023-12-18 | 2023-12-18 | Tracking method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311749451.0A CN117893567A (en) | 2023-12-18 | 2023-12-18 | Tracking method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117893567A true CN117893567A (en) | 2024-04-16 |
Family
ID=90651496
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311749451.0A Pending CN117893567A (en) | 2023-12-18 | 2023-12-18 | Tracking method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117893567A (en) |
-
2023
- 2023-12-18 CN CN202311749451.0A patent/CN117893567A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11710322B2 (en) | Surveillance information generation apparatus, imaging direction estimation apparatus, surveillance information generation method, imaging direction estimation method, and program | |
| US9270891B2 (en) | Estimation of panoramic camera orientation relative to a vehicle coordinate frame | |
| US20210056751A1 (en) | Photography-based 3d modeling system and method, and automatic 3d modeling apparatus and method | |
| EP4105766A1 (en) | Image display method and apparatus, and computer device and storage medium | |
| CN113029128B (en) | Visual navigation method and related device, mobile terminal and storage medium | |
| US20150002663A1 (en) | Systems and Methods for Generating Accurate Sensor Corrections Based on Video Input | |
| US9269004B2 (en) | Information processing terminal, information processing method, and program | |
| CN113361365B (en) | Positioning method, positioning device, positioning equipment and storage medium | |
| CN111798521A (en) | Calibration method, calibration device, storage medium and electronic equipment | |
| CN113610702B (en) | Picture construction method and device, electronic equipment and storage medium | |
| CN111583329B (en) | Augmented reality glasses display method and device, electronic equipment and storage medium | |
| CN112629565A (en) | Method, device and equipment for calibrating rotation relation between camera and inertial measurement unit | |
| CN117893567A (en) | Tracking method, device and storage medium | |
| CN113628284B (en) | Pose calibration data set generation method, device and system, electronic equipment and medium | |
| US9243935B2 (en) | Distance information estimating apparatus | |
| CN115205419A (en) | Instant positioning and map construction method and device, electronic equipment and readable storage medium | |
| CN113947621A (en) | Method and system for estimating displacement and three-dimensional scene point coordinates of multi-view camera | |
| CN108650465B (en) | Method and device for calculating augmented reality label of camera picture and electronic equipment | |
| JP2011154634A (en) | Image processing apparatus, method and program | |
| JP5662787B2 (en) | Mobile terminal and image processing method | |
| JP6906370B2 (en) | Image processing device | |
| CN113596336B (en) | Image acquisition method and device, electronic equipment and storage medium | |
| CA3102860C (en) | Photography-based 3d modeling system and method, and automatic 3d modeling apparatus and method | |
| CN102955942A (en) | Image positioning system and method of shot object | |
| CN117007078A (en) | Method, device, equipment, storage medium and product for estimating zero offset of gyroscope |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |