KR102050059B1 - Self-calibration apparatus and method of head mounted displays - Google Patents

Abstract

A method in which the head mounted display corrects a parameter for estimating its position, wherein when a user wearing the head mounted display moves based on guided movement information, sensors inserted into the head mounted display are moved according to the user's movement. Collect exercise data for. Randomly generate the parameters of the selected filter to estimate the position of the head mounted display, and determine whether the sum of the gaze changes calculated by the user's gaze movement using the motion data and the randomly generated parameters is less than the preset threshold. Check it. The generated parameter is adjusted until the sum of the gaze changes is smaller than the threshold, and if the sum of the gaze changes is smaller than the threshold, the adjusted parameter is set as a parameter for the selected filter.

Description

Self-calibration apparatus and method of head mounted displays

The present invention relates to an apparatus and method for self-calibration of a head mounted display.

Head Mounted Display (hereinafter referred to as 'HMD') uses an Inertial Measurement Unit (IMU) and an external positional tracker mounted inside the HMD to estimate the current position. Measure raw sensor data. The gyroscope, accelerometer, and geomagnetic field included in the inertial measurement unit calculates orientation.

Each sensor can be used to estimate HMD user posture, but because of its low precision, sensor fusion should be used. Sensor Fusion integrates multiple raw sensor data to compensate for the sensor's features and drawbacks, enabling accurate current posture estimation.

However, the data output from the inertial measurement device composed of MEMS (Micro Electro Mechanical System) forms an error due to various factors. To compensate for this problem, various sensor calibrations exist and can be classified into two types.

First, there is a method of calibrating a sensor of an inertial measurement device using an external device having high reliability. The other is to perform self-calibration using mathematical and kinematic relations without external devices.

The former can maximize the performance of the inertial measurement device, but a reliable device is not easy to use due to cost and weight. Therefore, in the case of IoT devices or mobile-based HMDs, since the environment and equipment used are different, there is a problem that the sensor cannot be always calibrated. Therefore, by using the posture estimation improvement technique through the self-correction, it is possible to output a high-precision posture even online.

Conventional self-calibration techniques include biasing to remove gyroscope drift, scaling between each axis of the accelerometer, eliminating external distortions of the geomagnetic field, and self-calibration using a camera. There is this. This improves the performance of the basic posture input to the sensor fusion and increases the precision of the estimated posture output after the sensor fusion.

Therefore, most self-calibration methods focus on improving the precision of raw sensor data. However, the importance of sensor fusion is also important in final posture estimation. Until now, there is no technique for improving sensor fusion performance using the self-calibration method.

Accordingly, the present invention provides an apparatus and method for self-calibrating a head mounted display capable of adjusting internal parameters without an external reference coordinate system.

As a method of correcting a parameter for estimating its position, a head mounted display which is one feature of the present invention for achieving the technical problem of the present invention,

When the user wearing the head mounted display moves based on the guided movement information, sensors inserted into the head mounted display collect movement data for the head mounted display according to the movement of the user; Randomly generating a parameter of the selected filter to estimate a position, and using the motion data and the randomly generated parameter, whether the sum of the gaze changes calculated by the gaze movement of the user is smaller than a preset threshold; Confirming, and adjusting the generated parameter until the sum of the gaze changes is less than a threshold, and setting the adjusted parameter as a parameter for the selected filter when the sum of the gaze changes is less than the threshold. It includes a step.

The step of determining whether it is smaller than the threshold value includes calculating a rotation angle of the head mounted display using the movement data and a random parameter, wherein the movement data includes acceleration, geomagnetic, and angular velocity information of the head mounted display. It may include.

After calculating the rotation angle, the method may include setting a viewpoint generated by the gaze of the user as a reference vector.

The reference vector may generate as a reference vector a vector of a viewpoint when the user is looking straight ahead after wearing the head mounted display.

After checking whether the sum is less than the threshold value, if the sum of the gaze changes is greater than the threshold value, the method may further include adjusting a randomly generated parameter.

An apparatus for correcting a parameter for estimating the position of the head mounted display is inserted into the head mounted display which is another feature of the present invention for achieving the technical problem of the present invention,

An exercise data collection module configured to collect exercise data of the head mounted display according to a user's movement when the user wearing the head mounted display moves based on previously guided movement information, and to estimate the position of the head mounted display from outside Receiving information on the selected filter, based on the parameter input module for randomly generating the parameters for the selected filter, and the workout data collected by the exercise data collection module and the parameters randomly generated by the parameter input module, the user Adjust the parameter until the sum of the gaze changes calculated from the gaze movements is less than a preset threshold; and if the sum of the gaze changes is less than the threshold, setting the adjusted parameter as a parameter for the selected filter. Farah Including adjusting the emitter module.

The parameter adjusting module may calculate a rotation angle of the head mounted display using the motion data and the random parameter.

The parameter adjusting module may set the converted vector as a reference vector when the user is looking straight ahead immediately after wearing the head mounted display.

The parameter adjusting module may adjust a randomly generated parameter when the sum of the line of sight changes is greater than the threshold value.

The movement data may include acceleration, geomagnetic, and angular velocity information of the head mounted display moved according to the movement of the user.

According to the present invention, internal parameters can be adjusted without an external reference coordinate system, and an internal parameter selection guide can be provided without signal analysis of an inertial measurement device.

In addition, the sensor can be easily calibrated compared to the existing sensor calibration, and can be applied to various sensor fusion methods.

1 is an exemplary view of a typical head mounted display.
2 is an exemplary diagram illustrating dynamic rms values according to general internal parameter changes.
3 is a structural diagram of a self-calibration apparatus of a head mounted display according to an embodiment of the present invention.
4 is an exemplary view of a user's gaze according to an exemplary embodiment of the present invention.
5 is an exemplary view of a motion gaze rotated according to a user's gaze according to an exemplary embodiment of the present invention.
6 is an exemplary diagram of calculation through the shortest distance according to an embodiment of the present invention.
7 is a flowchart illustrating an internal parameter adjusting method according to an embodiment of the present invention.
8 is a diagram illustrating input of guided exercise data according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise.

Hereinafter, a self-calibrating apparatus and a method of a head mounted display according to an embodiment of the present invention will be described with reference to the drawings. Before describing an exemplary embodiment of the present invention, a general head mounted display and a root mean square (RMS) according to internal parameter changes will be described with reference to FIGS. 1 and 2.

1 is an exemplary diagram of a general head mounted display, and FIG. 2 is an exemplary diagram illustrating dynamic rms values according to general internal parameter changes.

First, as shown in FIG. 1, in order to estimate a posture of a user who is currently wearing the HMD, raw sensor data using the inertial measurement device 10 and the external motion tracker 20 mounted inside the HMD worn by the user. Measure The inertial measurement device 10 includes sensors such as a gyroscope, an accelerometer, a geomagnetic field, and these sensors calculate a posture.

Each sensor can be used to estimate the posture of a user wearing an HMD, but because of its low precision, the sensor fusion 30 can be used to fuse raw sensor data to compensate for sensor features and shortcomings, Posture must be estimated.

The sensor fusion 30 compensates for errors occurring in each sensor included in the HMD 10, and serves to accurately estimate a user's posture. Therefore, it can be seen that the sensor fusion 30 includes a function of correcting raw sensor data.

However, since the final posture of the user is the result of the sensor fusion 30, improving the accuracy of the sensor fusion 30 is also a very important topic. The sensor fusion 30 has several filters 30-1 to 30-4 having various characteristics. In addition, there are always parameters that determine performance in various sensor fusion methods.

For example, in the complementary filter 30-1, there is α that determines the update rate of the gyroscope and the accelerometer. In the case of the Kalman filter 30-2, a covariance value for prediction and measurement is required. In addition, a gain value β exists in the Magdwick filter 30-4.

The importance of the sensor fusion parameters present in each of these filters is described with reference to FIG. 2. 2 shows the results of an external reference coordinate system and sensor fusion results in root mean square (RMS).

In general, the posture comparison of a user is divided into a static rate (Rate Monotonic Scheduling) comparing the angles of the stationary state and a dynamic RMS comparing the rotational motion for a certain time. Units representing static and dynamic RMS are expressed in degrees or radians.

The reference coordinate system of the result shown in FIG. 2 is an optical encoder of a rotary platform, and the inertial measurement apparatus 10 used for the comparison object is MPU-6500 used for Oculus DK2. The raw sensor data output from the inertial measurement unit was inputted to the complementary filter 30-1 and the Mazwick filter 30-4. The internal parameter values α and β set at this time were arbitrarily input.

As shown in FIG. 2, it can be seen that the sensor fusion accuracy is significantly changed depending on the determination of the parameter. Thus, the results shown in FIG. 2 show how important parameter selection is in sensor fusion 30.

A general method of determining internal parameters in the sensor fusion 30 is to compare with an external precision reference system. The error function is set by the difference between the external precision reference system and the estimated value, and the parameter is obtained as the optimal solution of the error function and set as the sensor fusion parameter. In order to determine the internal parameters using this method, there must be a precise reference coordinate system, and it must be a form that can be measured with the HMD.

Alternatively, there is a guide for setting each parameter. Α of the complementary filter 30-1 is a cut-off frequency, and the Kalman filter 30-2 measures the covariance of the data measured by the inertial measurement apparatus 10 and the covariance of the data when predicted. it means. In addition, β of the Maeswick filter 30-4 is a reflection ratio of the gradient-decent method.

These methods directly acquire raw sensor data from sensors included in each inertial measurement device 10 and require signal analysis and calculation processes. In addition, the criteria for selecting parameters were not precisely formulated. Therefore, the user or developer must determine the trend of the data and make an empirical decision.

And since the optimum parameter changes according to the tendency of the exercise, it is not easy to select the parameter without external reference. In particular, the more various exercise environments such as HMD and IoT devices, the harder it is to set the parameters of the sensor fusion 30. Therefore, in the case of using the same inertial measurement apparatus 10, the public parameter is often borrowed. This can also adversely affect posture estimation if the same sensor operates in different environments.

Accordingly, an embodiment of the present invention proposes a self-calibration apparatus and a self-calibration method using the same, which can guide the sensor fusion parameter setting as in the conventional self-calibration technique. In the embodiment of the present invention, the self-calibration device 100 is included in the HMD 10 for convenience of description, and the self-calibration device 100 collects exercise data or processes the sensor fusion 30. The description is omitted.

3 is a structural diagram of a self-calibration apparatus of a head mounted display according to an embodiment of the present invention.

As shown in FIG. 3, the self-calibration apparatus 100 according to the embodiment of the present invention is controlled by at least one processor, and includes the exercise data collection module 110, the parameter input module 120, and the parameter adjustment module ( 130).

The exercise data collection module 110 receives the exercise data collected by the inertial measurement apparatus 10 included in the HMD worn by the user and including a plurality of sensors. Here, the movement data is rotation information obtained by the inertial measurement apparatus 10 when the user moves based on the guide information provided through the HMD worn by the user, and includes three-axis acceleration, geomagnetic, and angular velocity information.

Here, the contents related to the exercise data received by the exercise data collection module 110 will be described first with reference to FIGS. 4 and 5.

4 is an exemplary view of a user's gaze according to an exemplary embodiment of the present invention, and FIG. 5 is an exemplary view of an exercise gaze rotated according to a user's gaze according to an exemplary embodiment of the present invention.

In general, the conventional self-calibration technique sets the error function in consideration of the physical positional relationship of the user or the mechanical position worn by the user. Through this, the correction coefficient of each sensor is optimized. With this in mind, an embodiment of the present invention intends to introduce a new error function that can optimize the parameters of the sensor fusion 30 without an external reference coordinate system.

The filters used for the sensor fusion 30 vary and the internal parameter dimension also varies depending on the type of filter. For example, in the case of the complementary filter 30-1, since the α for three axes can be substituted, the parameter is three-dimensional. On the other hand, in the case of the extended Kalman filter with the observable performance among the Kalman filters 30-3, the two covariance matrices are parametric and are total six dimensions.

Finally, the Maeswick filter 30-4 has only β and is divided into two dimensions in some cases. Because of these different fusion methods and the dimensions of the parameters, it is difficult to select the error function of self-calibration by internal rules. Therefore, the embodiment of the present invention proposes a method in which the inertial measurement device 10 guides and estimates the external movement to follow the tendency of the movement.

4A illustrates a model of the user's gaze when the user wears the HMD in the virtual sphere ①. As shown in (a) of FIG. 4, it is assumed that the point (③) where the virtual sphere meets the dotted line arrow (②) based on the user is the trace of the user's gaze.

Then, as shown in (b) of Figure 4, the plurality of points (4) is the trace of the time left in the virtual sphere while the actual user moves. Since the physical movement of the HMD worn by the user varies depending on the content used, the distribution of traces left in the virtual sphere (1) is also different.

If the user assumes that the user is looking at the front and the user intentionally rotates only in the z-axis direction (yaw direction), the trace (⑤) of the horizontal line of sight is generated as shown in FIG. do. Also, if the user rotates only in the y-axis direction (pitch direction), which is the second direction, as shown in FIG. 5B, a trace ⑥ of the vertical line of sight is generated.

In addition, if the user rotates only in a third direction (roll direction) with respect to an axis in a horizontal plane parallel to the moving direction, the line of sight is also referred to as a point (hereinafter, referred to as a 'viewpoint') as shown in FIG. ) And a fixed trace (⑦) is generated. At this time, the trace generated in the virtual sphere is an ideal case, no matter how the user moves based on the guide information, it is difficult to generate a straight line as shown in (a) to (c) of FIG. 5.

However, if the user wearing the HMD performs the guide movement, the movement of the gaze traces in all three directions has the meaning of the shortest distance. Using this meaning, we set up the error function to be used for internal parameter optimization.

Meanwhile, the parameter input module 120 of FIG. 3 randomly generates a parameter corresponding to the selected filter information input from the outside. Here, the method of randomly generating the parameter or the type of the generated parameter is not limited to either.

The parameter adjusting module 130 calculates a rotation angle of the HMD using the workout data received from the workout data collection module 110. Through this, the parameter adjusting module 130 may know how much the HMD is rotated. Here, the method of calculating the rotation angle of the HMD from the motion data by the parameter adjustment module 130 may be performed in various ways, and thus, detailed descriptions thereof will be omitted.

The parameter adjusting module 130 calculates the rotation angle of the HMD and then generates a reference vector using the viewpoint of the user's eyes. As shown in (a) of FIG. 4, the reference vector refers to a vector generated based on a viewpoint in which the user's gaze appears in a virtual three-dimensional space when the user first wears the HMD and faces the front.

The parameter adjusting module 130 generates a reference vector and then calculates the movement of the user's gaze. The parameter adjusting module 130 measures the gaze change based on the calculated movement of the gaze of the user, and adjusts the parameter until the sum of the function tolerences is smaller than a preset threshold. When the sum of gaze changes is smaller than the threshold value, the parameters of the filter used to determine the rotation angle of the HMD are output as optimized parameters.

In this case, an example in which the parameter adjusting module 130 performs the calculation through the shortest distance to calculate the sum of the line of sight changes will be described with reference to FIG. 6.

6 is an exemplary diagram of calculation through the shortest distance according to an embodiment of the present invention.

As shown in FIG. 6, the virtual space must faithfully reflect the physical motion, so the sum of the traces represented by the guided motion also draws the shortest distance.

Circle (1) shown in (b) of FIG. 6 means a virtual sphere, and each black dot (8) represents the trace of the gaze of each sampling time. That is, when the user moves based on the guided movement, the raw sensor data, which is data about the movement, is input, and the sensor fused posture should draw the trace as shown in FIG. 5 in the virtual space.

Xn shown in FIG. 6 is a spatial coordinate of the trace of arbitrary nth gaze. The imaginary sphere is a unit sphere with a distance of 1 from the origin. Therefore, after fixing the reference point as the reference point to (1, 0, 0), the nth estimated posture can be calculated by the following equation (1).

Therefore, ln means the physical distance between Xn and Xn + 1, the sum of these means the sum of the line of sight change in the guided motion. Here, R is a rotation matrix indicating when the HMD rotates in the order of (ψ, θ, φ). Multiply this value by the respective position coordinates to represent the coordinate system after rotation.

 This value acts as an error function, which enables parameter optimization. Optimization is described using an example of gradient descent, but is not necessarily limited to this. Equation 2 is an equation for optimization.

Here, P _k means a parameter of the sensor fusion to be used. The parameters vary in dimension depending on the filter used, and are initially generated randomly by the parameter input module 120 according to the selected filter information received from the outside.

f (P _k ) means the sum of eye changes calculated by inputting parameters using guided motion data. Α _k represents the reflectance ratio of the gradient, and ε means the threshold value of the repetition. Use this to optimize the parameters.

Here, the threshold of repetition is set to different values according to the method selected to optimize the parameter. In the embodiment of the present invention, the use of gradient descent as a parameter optimization method will be described as an example. Therefore, it is described by using 0.001 as a threshold value of repetition.

Next, a method of adjusting the parameter of the selected filter using the above-described self-calibration apparatus 100 will be described with reference to FIGS. 7 and 8.

7 is a flowchart illustrating an internal parameter adjusting method according to an embodiment of the present invention. 8 is a diagram illustrating input of guided exercise data according to an embodiment of the present invention.

As shown in FIG. 7, when the user rotates or moves on the basis of the guide head motion pre-stored in the HMD, the self-calibration apparatus 100 receives movement data regarding the moved HMD (S100). The received motion data includes acceleration, geomagnetic, and angular velocity information.

And in the embodiment of the present invention will be described with an example that the guide head motion is provided to show a constant movement for 8 to 10 seconds. The guide head motion is pre-stored in the HMD, as shown in FIG. 8. The guide head motion will first be described with reference to FIG. 8.

As shown in FIG. 8, it is ideally preferable that the HMD rotates vertically and horizontally on each axis when generating the movement data. However, the rotation of the HMD is the rotation of the person wearing the HMD, and thus may not be accurate.

Therefore, in the embodiment of the present invention will be described by rotating the HMD by an external physical force without wearing by the user as an example. When generating the motion data by the rotation of the HMD, the data is generated by rotating three wheels in the yaw direction, three wheels in the pitch direction, and three wheels in the roll direction. The order of the orientations is irrelevant, but as mentioned, the user must consider precision in physical rotation.

Meanwhile, referring to FIG. 7, when the self-calibration apparatus 100 receives information about a selected filter of a sensor fusion method to be used from the outside, the magnetic correction apparatus 100 randomly generates a parameter corresponding to the filter (S101 and S102). Since the method of randomly generating the parameter corresponding to the filter by the self-calibration apparatus 100 may be performed in various ways, the embodiment of the present invention is not limited to any one method.

Then, using the random parameter generated in step S102, the sum of the gaze changes with respect to the gaze of the user wearing the HMD is calculated. The process of calculating the sum of eye changes is as follows.

First, the self-calibration apparatus 100 calculates a rotation angle indicating how much the HMD is rotated by using the workout data received in step S100 and the random parameter generated in step S102 (S103). The self-calibration apparatus 100 sets the viewpoint by the line of sight of the user wearing the HMD in a three-dimensional space as a vector (S104). In the embodiment of the present invention, as shown in (a) of FIG. 4, the setting of the vector when the user is facing the front as a reference vector is described as an example, but is not necessarily limited thereto.

When the reference vector is generated through the step S104, the self-calibration apparatus 100 calculates the sum of the change of the gaze by calculating the movement of the gaze of the user (S105). Then, the self-calibration apparatus 100 checks whether the sum of the calculated changes in the line of sight is smaller than a preset threshold value (S106).

If the sum of the calculated gaze changes is greater than the threshold value, the self-calibration apparatus 100 adjusts the parameter (S108) and then performs the procedure after step S103. However, if the sum of gaze changes is less than the threshold value, the parameter that minimizes the sum of gaze changes is output as a parameter optimized for the selected filter (S107).

In other words, the self-calibration apparatus 100 outputs an optimized parameter when the sum of the gaze changes no longer changes while adjusting the parameter so that the sum of the gaze changes becomes smaller. This improves the HMD's position estimation performance with optimized parameters without expensive references.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (10)

A method of calibrating a parameter for a head mounted display to estimate its position,
When the user wearing the head mounted display moves based on guided movement information, the head mounted display collects movement data of the head mounted display by sensors inserted into the head mounted display according to the movement of the user;
The head mounted display randomly generating a parameter of a filter selected from the outside to estimate its position;
The rotation angle of the head mounted display is calculated using the movement data and the random parameter generated randomly, and the sum of the gaze changes obtained by calculating the gaze movement of the user using the random parameter is used to adjust the random parameter. Verifying that the threshold is less than the preset threshold, and
Adjusting the generated random parameter until the sum of the gaze changes is less than a threshold value, and setting the adjusted random parameter as a parameter for the selected filter when the sum of the gaze changes is less than the threshold value.
Self-calibration method comprising a. The method of claim 1,
And the motion data includes acceleration, geomagnetism, and angular velocity information of the head mounted display. The method of claim 2,
After calculating the rotation angle,
Setting a viewpoint generated by the gaze of the user as a reference vector
Self-correction method further comprising. The method of claim 3,
The reference vector is a self-calibration method of generating a vector of a view point when the user is looking directly to the front immediately after wearing the head mounted display. The method of claim 3,
After checking whether it is less than the threshold,
Adjusting a randomly generated parameter when the sum of the gaze changes is greater than the threshold value.
Self-correction method further comprising. A self-calibration device inserted into a head mounted display to calibrate a parameter for estimating the position of the head mounted display.
An exercise data collection module configured to collect exercise data of the head mounted display according to a user's movement when a user wearing the head mounted display moves based on previously guided movement information;
A parameter input module for randomly generating a parameter for an externally selected filter to estimate the position of the head mounted display, and
Based on the workout data collected by the workout data collection module and a random parameter randomly generated by the parameter input module, a sum of gaze changes calculated by the gaze movement of the user is preset to adjust the random parameter. The random parameter is adjusted until it is smaller, and the rotation angle of the head mounted display is calculated using the motion parameter and the adjusted random parameter, and the adjusted random value is smaller than the threshold value. A parameter adjustment module for setting a parameter as a parameter for the selected filter
Self-calibration device comprising a.
delete The method of claim 6,
The parameter adjustment module,
The self-calibration device of setting the converted vector as a reference vector at the point in time when the user is looking straight ahead after wearing the head mounted display. The method of claim 8,
The parameter adjustment module,
And adjusting the randomly generated parameter when the sum of the gaze changes is greater than the threshold value. The method of claim 6,
And the motion data includes acceleration, geomagnetic, and angular velocity information of the head mounted display moved according to the movement of the user.

Images