KR20190098003A - Method for estimating pose of device and thereof - Google Patents

Abstract

Provided are a method for estimating a posture of a device and a device thereof. To estimate a posture of the device, the method generates IMU data of the device, determines a posture at a current time point based on the IMU data, and estimates a posture at a future time point using a current predicted motional state array generated based on the current posture and IMU data.

Description

Posture estimation method of a device and its device {METHOD FOR ESTIMATING POSE OF DEVICE AND THEREOF}

The following embodiments relate to a method and apparatus for estimating the attitude of the apparatus, and more particularly, to a method and apparatus for estimating the attitude of the apparatus using an inertial measurement apparatus (IMU).

Display technologies such as "virtual reality (VR)" or "augmented reality (AR)" should be felt by the user as if the output image is real. These technologies allow the user to experience a variety of experiences that cannot be experienced in reality. Images provided to the user through VR or AR are important for real time. In particular, the output image closely related to the posture of the user should accurately detect the posture of the user. If the real time of the output image is low, the user may be uncomfortable due to an error occurring in the difference between the posture of the user and the output image.

For example, a device worn by the user may be used to detect the posture of the user. Tight coupling method of detecting the attitude of the device through many sensors has the advantage of high accuracy, but due to a lot of data, there are many constraints to consider and the calculation speed is slow and delay occurs.

One embodiment may provide a method of estimating the location of a device and the device.

One embodiment can provide a method and apparatus for estimating the position of a device using a loose coupling method.

According to an aspect, a method of estimating a posture of a device performed by a device may include generating IMU data using an Inertial Measurement Unit (IMU) of the device, based on the IMU data. Determining a first pose, generating a present prediction motional state array based on the IMU data, and based on the current predicted motion state array, Estimating an M-th prediction pose at a M-th point after the first time point.

The generating of the current predicted motion state arrangement may include smoothing the IMU data and generating a first motional state vector of the first view based on the flattened IMU data. Determining an Nth predicted motion state vector of the Nth view based on the first motion state vector, and a motion model, a previous prediction motional state array, and the Nth view. And generating the current predicted exercise state array based on a predicted exercise state vector.

The generating of the first motion state vector may include generating the first motion state vector by correcting the flattened IMU data based on a difference between the first pose and a first estimated pose. It may include the step.

The determining of the N th predicted motion state vector may include: determining the N th predicted motion state vector using a recurrent neural network of a Long-Short Term Memory (LSTM) method using the first motion state vector as an input. Generating an exercise state vector.

The generating of the first motion state vector may include generating the first motion state vector based on at least one of an ocular movement velocity of a user of the device and a head movement velocity of the user. And determining the frequency of doing so.

The attitude estimation method further includes determining whether the state of the device is a static state based on the IMU data, and determining the first attitude of the device comprises: determining whether the state of the device is a static state The method may include determining a first posture based on whether or not the posture is determined.

The attitude estimation method may further include generating a virtual object to be output at the Mth time point, and outputting the virtual object.

The posture estimating method may further include adjusting the virtual object based on a change in posture of the device occurring during the rendering of the virtual object.

The outputting of the virtual object may include outputting the virtual object such that the image for the Mth viewpoint and the virtual object are synthesized.

The virtual object may be a holographic image.

The pose estimation method may include generating a map of the surrounding space of the device, determining an initial pose using the map, and based on IMU data, the device is in a static state. Determining whether or not to recognize, wherein the first posture may be determined when the device is in a static state.

The generating of the map may include generating a left image using the left camera of the device, generating one or more first feature points from the left image, and generating a right image using the right camera of the device. Generating one or more second feature points from the right image, matching the first feature points and the second feature points, and matching and not matching one of the first feature points and the second feature points. And generating the map based on feature points that are not.

The determining of the initial posture may include generating one or more feature points from an image generated by at least one of the left camera and the right camera, and a candidate feature point preset in the map. The method may include determining a target point corresponding to the one or more feature points, and determining the initial posture based on the target feature point.

The determining of the initial posture may include determining a remaining feature point that does not correspond to candidate feature points preset in the map among the one or more feature points, and the feature point corresponding to the remaining feature points. The method may further include updating the map by adding to.

The device may be a wearable device.

According to another aspect, an apparatus for estimating a pose includes at least one camera, an inertial measurement unit (IMU), a memory in which a program for estimating a pose is recorded, and a processor for performing the program, The program may include generating IMU data using an Inertial Measurement Unit (IMU) of the device, determining a first pose of the device based on the IMU data, and the IMU data. Generating a current predicted exercise state arrangement based on the second predictive state, and estimating an M-th predicted posture at an M-th time point after a first time point of the first posture based on the current predicted exercise state arrangement. .

1 illustrates images in which a virtual object is output according to an example.
2 is a block diagram of an apparatus according to an embodiment.
3 is a flowchart illustrating a cluster optimization method for attitude estimation of an apparatus according to an example.
4 is a flowchart of a first cluster optimization method according to an example.
5 is a flowchart of a method of generating a map for a peripheral space of a device according to an example.
6 is a flowchart of a method of determining an initial posture using a map according to an example.
7 illustrates a method of optimizing a cluster while determining an initial posture according to an example.
8 is a flowchart of a method of updating a map, according to an example.
9 illustrates a second cluster optimization method according to an example.
10 is a flowchart of a method of estimating an M-th prediction attitude through second cluster optimization, according to an embodiment.
11 is a flowchart of a method of generating a current predicted exercise state arrangement according to an example.
12 illustrates a method of estimating an attitude of an i + x point of view based on an attitude of an i th view according to an example.
13 illustrates a method of calculating a current predicted exercise state arrangement according to an example.
14 illustrates a method of detecting a static state according to an example.
15 illustrates a result due to jitter generated by circular detection according to an example.
16 is a flowchart of a method of outputting a virtual object according to an example.
17 illustrates a method of outputting a composite image based on an estimated attitude of the apparatus according to an example.
18 is a block diagram illustrating modules of an apparatus according to an example.
19 is a view illustrating a pose estimation result according to various detection methods according to an example.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference numerals in the drawings denote like elements.

Various modifications may be made to the embodiments described below. The examples described below are not intended to be limited to the embodiments and should be understood to include all modifications, equivalents, and substitutes for them.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of examples. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

In addition, in the description with reference to the accompanying drawings, the same components regardless of reference numerals will be given the same reference numerals and duplicate description thereof will be omitted. In the following description of the embodiment, when it is determined that the detailed description of the related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 illustrates images in which a virtual object is output according to an example.

For example, the first image 110 includes a real object 111 and a virtual object 112. The virtual object 112 may be additional information about the real object 111. The position of the virtual object 112 in the first image 11 may be determined based on the gaze of the user and the position of the real object 111.

When the user's gaze changes, the second image 120 changes. The second image 120 includes a real object 121 corresponding to the real object 110 and a virtual object 122 corresponding to the virtual object 112. The position of the actual object 121 in the second image 120 changes immediately to reflect the user's gaze. However, when the gaze of the user is not immediately detected, the position of the virtual object 122 in the second image 120 is not changed and may be determined to be the same as the position of the virtual object 112. Accordingly, the relative position between the real object 121 and the virtual object 122 appearing in the second image 120 may not be appropriate.

According to one aspect, the gaze of the user may be detected by determining a posture of the device corresponding to the gaze of the user. The attitude of the device may be determined or estimated through the sensors of the device. Hereinafter, a method of estimating the attitude of the apparatus will be described in detail with reference to FIGS. 2 to 18.

The term "posture" below means the current posture determined using the currently measured data, and the term "predictive posture" means the future posture estimated using the currently measured data.

2 is a block diagram of an apparatus according to an embodiment.

The apparatus 200 includes a communication unit 210, a processor 220, a memory 230, a camera 240, and an inertial measurement unit (IMU). The device 200 may be an electronic device. For example, the device 200 may be a device that outputs an image for providing AR or VR to a user. According to an aspect, the device 200 may be a wearable device. For example, the device 200 may be in the form of glasses or a head mounted display (HMD), but is not limited thereto. According to another aspect, the device 200 may be used to measure the location of the user, estimate the location of the user, and the like. According to another aspect, the device 200 may be used for autonomous driving.

The communication unit 210 is connected to the processor 220, the memory 230, the camera 240, and the IMU 250 to transmit and receive data. The communication unit 210 may be connected to another external device to transmit and receive data.

The communication unit 210 may be implemented as a circuit in the device 200. For example, the communication unit 210 may include an internal bus and an external bus. As another example, the communication unit 210 may be an element connecting the device 200 to an external device. The communication unit 210 may be an interface. The communication unit 210 may receive data from an external device and transmit data to the processor 220 and the memory 230.

The processor 220 processes data received by the communication unit 210 and data stored in the memory 230. A "processor" may be a data processing device implemented in hardware having circuitry having a physical structure for performing desired operations. For example, desired operations may include code or instructions included in a program. For example, data processing devices implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , An application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

The processor 220 executes computer readable code (eg, software) stored in the memory (eg, memory 230) and instructions caused by the processor 220.

The memory 230 stores data received by the communication unit 210 and data processed by the processor 220. For example, the memory 230 may store a program. The stored program may be a set of syntaxes that are coded to estimate the attitude of the device and executable by the processor 220.

According to one aspect, the memory 230 may include one or more volatile memory, nonvolatile memory and random access memory (RAM), flash memory, hard disk drive, and optical disk drive.

The memory 230 stores a set of instructions (eg, software) for operating the device 200. The instruction set for operating the device 200 is executed by the processor 220.

The camera 240 generates an image by photographing a scene. For example, the camera 240 may include a left camera that generates a left image to be output to the user's left eye and a right camera that generates a right image to be output to the user's right eye. There is a binocular parallax between the left image and the right image. As another example, the camera 240 may generate one image, and the processor 220 may generate a left image and a right image using an image processing method such as image warping.

The IMU 250 may generate IMU data by measuring acceleration and angular velocity according to the movement of the device 200.

Although not shown, the device 200 may include sensors capable of measuring the position or posture of the device 200. For example, device 200 may include a global positioning system (GPS) and is not limited to the described embodiments.

The communication unit 210, the processor 220, the memory 230, the camera 240, and the IMU 250 will be described in detail with reference to FIGS. 3 to 18 below.

3 is a flowchart illustrating a cluster optimization method for attitude estimation of an apparatus according to an example.

According to one aspect, the first cluster optimization step 310 and the second cluster optimization step 320 may be performed to estimate the attitude of the apparatus 200. Cluster optimization means optimizing the parameters and algorithms used to estimate the attitude of the device 200.

The first cluster optimization step 310 may be performed using a tight coupling cluster optimization scheme. The tight coupling cluster optimization method estimates the attitude of the device 200 using the data measured using a large number of sensors. For example, an image generated using the camera 240 and IMU data generated using the IMU 250 may be used to estimate the attitude of the device 200. The first cluster optimization step 310 is described in detail with reference to FIGS. 4 to 8 below.

The second cluster optimization step 320 may be performed using a loose coupling cluster optimization scheme. The loose coupling cluster optimization method estimates the attitude of the device 200 using the data measured using a small number of sensors. For example, only IMU data generated using IMU 250 may be used to estimate the attitude of device 200. Since the IMU data includes the acceleration and the angular velocity of the device 200, the current attitude may be determined when integrating the IMU data and overlapping the integrated value with the previous attitude.

 The less data used to estimate the posture, the faster the posture can be estimated. A second cluster optimization step 320 is described in detail below with reference to FIGS. 9-15.

4 is a flowchart of a first cluster optimization method according to an example.

Step 310 described above with reference to FIG. 3 includes steps 410 through 440 below.

In operation 410, the device 200 generates a map of the surrounding space of the device 200. The map may be generated using the images captured by the camera 240 of the device 200. A method of generating a map is described in detail with reference to FIG. 5 below.

In operation 420, the device 200 determines an initial pose of the device 200 using the map. A method of determining an initial posture is described in detail with reference to FIG. 6 below.

In step 430, the device 200 determines whether the state of the device 200 is a static condition based on the IMU data. The static state means that there is no movement of the device 200 or a movement below a preset threshold appears.

The IMU data typically contains noise, which causes errors in the guess pose of the device 200. When the posture is estimated using the static state (or the stationary state) of the device 200, the influence of noise may be reduced. By analyzing local variations of the IMU data, it may be detected whether the device 200 is in a static state. If the device 200 is stationary, then constraints are added to increase the accuracy of attitude estimation when performing cluster optimization at the next point in time. Constraints are as follows:

i) The location of the device 200 may be obtained based on the spatial location of the map point (eg, three-dimensional location) and the location of the feature point of the image (left image / or right image) corresponding thereto. ii) The location of the new map point in space may be calculated by the location of the device 200 and the feature point corresponding to the new map point on the image. iii) The position and attitude of the device between the before and after images can be calculated through the IMU data and the IMU parameters between the before and after images. The IMU parameters may be a workout state vector. iv) IMU parameters may be calculated using IMU data between a plurality of images and a pose of the device 200 corresponding to the plurality of images. v) Through the detected still state, it can be determined that the posture between two images changes to zero.

Since there is a constraint between the map point, the attitude of the device 200 and the IMU data, the attitude of the device 200 that satisfies this constraint as much as possible may be determined. The more the number of the above constraints is satisfied, the more accurate attitude can be estimated, but the estimation speed is slower.

In step 440, it is determined whether the first cluster optimization or the second cluster optimization is performed according to the state of the device 200.

Through the first cluster optimization step 310, parameters and algorithms to be used for attitude estimation in the second cluster optimization step 320 may be optimized. Parameters and algorithms to be used for attitude estimation in the second cluster optimization step 320 are described in detail with reference to FIGS. 10 to 13 below.

5 is a flowchart of a method of generating a map for a peripheral space of a device according to an example.

Step 410 described above with reference to FIG. 4 includes steps 510-540 below.

In operation 510, the device 200 may generate a left image. For example, the left camera is used to generate the left image. As another example, the device 200 may generate a left image by warping a reference image photographed using a camera.

In step 515, device 200 generates one or more first feature points in the generated left image. The feature points can be edges and textures in the image. For example, device 200 may use methods such as brisk and sift to generate feature points.

In operation 520, the device 200 may generate a right image. For example, the right camera is used to generate the right image. As another example, the device 200 may generate a right image by warping a reference image photographed using a camera.

In step 525, device 200 generates one or more second feature points in the generated right image.

In step 530, the device 200 matches the first feature point and the second feature point. For example, the device 200 may verify a matching relationship and remove erroneous matching using a fitting via random sample consensus (RANSAC) method.

In operation 540, the device 200 generates a map based on the matched and non-matched feature points of the first and second feature points.

6 is a flowchart of a method of determining an initial posture using a map according to an example.

Step 420 described above with reference to FIG. 4 includes steps 610 through 630 below.

In step 610, the device 200 generates one or more feature points from the image generated by the camera.

In operation 620, the device 200 determines a target feature point corresponding to one or more feature points among candidate feature points preset in the map.

In step 630, the device 200 determines an initial pose of the device 200 based on the target feature point. For example, the initial pose may be the current pose of the camera 240 (or the device 200) corresponding to the gaze of the user. The posture may mean a view point of the camera.

While step 310 is repeatedly performed, parameters for estimating the future pose of the device 200 may be optimized. The parameters optimized through step 310 may be used continuously in step 320.

7 illustrates a method of optimizing a cluster while determining an initial posture according to an example.

The first pose 710 of the apparatus 200 is determined by using the image and the map photographed in the first pose 710. For example, target feature points 721 and 722 of the map may be determined based on the image photographed in the first posture 710. The user may wear and move the device 200. While moving, the athletic state of the device 200 may be detected. The athletic condition may be IMU data. The posture of the device 200 after the first posture 710 may be estimated based on the determined first posture 710 and the IMU data. Noise-filtered IMU data may be used to reduce the error of the estimated pose. For example, noise biased IMU data may be used to estimate attitude.

The second posture 711 of the device 200 is determined by using the image and the map photographed in the second posture 711. For example, target feature points 721, 722, 723, 724, and 725 of the map may be determined based on the image photographed in the second posture 711. The posture of the device 200 after the second posture 711 may be estimated based on the determined second posture 711 and the IMU data.

The third pose 712 of the apparatus 200 is determined by using the image and the map photographed in the third pose 712. For example, target feature points 723, 725, 726, 728, and 729 of the map may be determined based on the image photographed in the third posture 712. The posture of the device 200 after the third posture 712 may be estimated based on the determined third posture 712 and the IMU data.

The fourth posture 713 of the apparatus 200 is determined using the image and the map photographed in the fourth posture 713. For example, target feature points 726, 727, 728, and 729 of the map may be determined based on the image photographed in the fourth posture 713. The posture of the device 200 after the fourth posture 713 may be estimated based on the determined fourth posture 712 and the IMU data.

The first postures 710 to 713 may be determined as actual postures, and parameters for estimating postures based on the actual postures may be adjusted. By iterative estimation, the error between the actual pose and the estimated pose may be reduced.

Each of the first postures 710 to 713 may be an initial posture, and it may be understood that the initial posture is updated for the second cluster optimization.

8 is a flowchart of a method of updating a map, according to an example.

Step 420 described above with reference to FIG. 4 may further include steps 810 and 820 below.

In operation 810, the apparatus 200 determines a remaining feature point that does not correspond to candidate feature points preset in the map among one or more feature points of the generated image.

In step 810, the device 200 updates the map by adding feature points corresponding to the remaining feature points to the map. The added feature point becomes a candidate feature point. As the candidate feature points increase, the accuracy of the attitude of the device 200 determined using the feature points may increase.

9 illustrates a second cluster optimization method according to an example.

The second cluster optimization method using the loose coupling cluster optimization method is a method of quickly estimating the attitude of the apparatus 200 using data measured by a small number of sensors. For example, fewer data may be used than data used in the first cluster optimization method. By using a small number of data, the attitude estimation speed of the device 200 may be increased.

According to one aspect, the current posture of the device 200 may be determined using the IMU data, and the posture of the future device 200 may be estimated using the history of the IMU data. Circular detection may be used to reduce the difference between the current pose and the estimated pose.

Hereinafter, a method of estimating the attitude of the apparatus 200 using the loose coupling cluster optimization scheme will be described in detail with reference to FIGS. 10 to 15.

10 is a flowchart of a method of estimating an M-th prediction attitude through second cluster optimization, according to an embodiment.

Step 320 described above with reference to FIG. 3 may include steps 1010 to 1040 below. According to one aspect, step 320 may be performed when the state of the device 200 is in a static state.

In step 1010, the device 200 generates IMU data using the IMU 250.

In step 1020, the device 200 determines a first pose of the device 200 based on the IMU data. The first posture is the posture of the device 200 at the present time (i time point).

According to one aspect, the device 200 may determine the first posture based on whether the state of the device 200 is a static state. For example, when the device 200 is in a static state, the device 200 determines the first posture using the IMU data.

In step 1030, the device 200 generates a current predicted workout state arrangement based on the IMU data. The term “predictive athletic state arrangement” means a list of vectors predicting the athletic state of the device 200 according to time points. The term " present prediction motional state array " means the predicted motion state array estimated at the current point in time (i point). The term "previous prediction motional state array" means a predicted motion state array estimated at a previous time point (i-1 time point). That is, the current prediction exercise state arrangement may be generated by updating the previous prediction exercise state arrangement.

A method for generating a current predicted athletic state arrangement is described in detail below with reference to FIGS. 11-13.

In operation 1040, the device 200 estimates the M-th predicted posture at the M-th point after the first posture based on the current predicted athletic state arrangement. M is a natural number greater than one.

11 is a flowchart of a method of generating a current predicted exercise state arrangement according to an example.

Step 1030 described above with reference to FIG. 10 includes the following steps 1110 to 1140.

In step 1110, the device 200 smoothes the measured IMU data to remove noise and outliers. For example, a Kalman filter can be used to smooth the IMU data.

In step 1120, the device 200 generates a first motion state vector of the first time point based on the flattened IMU data. For example, the exercise state vector may be expressed by Equation 1 below.

In Equation 1, S is a motion state vector, v is a motion speed, a _v is a motion acceleration, w is a turning speed, and a _w is a turning acceleration. Swing speed and swing acceleration can be added to indicate each of the six axis directions. In addition, the exercise state vector may further include a position of the camera 240.

According to an aspect, the device 200 may generate the first exercise state vector by correcting the flattened IMU data based on a difference between a first pose and a first estimated pose that is previously estimated. . The difference between the first posture and the first estimated posture estimated in advance may be defined as the supplemental amount.

According to one aspect, the device 200 generates a first motion state vector based on at least one of an ocular movement velocity of a user of the device 200 and a head movement velocity of the user. Frequency can be determined. For example, when compensating for differences caused by purifying detection, the replenishment rate is associated with the user's eye movement rate and / or the user's head movement rate. The faster the hair movement, the faster the replenishment.

In operation 1130, the device 200 determines the Nth athletic state vector at the Nth time point based on the first athletic state vector. For example, the Nth view may be a view of the farthest view that can be predicted from the current view. N is a natural number larger than M.

According to an aspect, the device 200 generates an N-th predicted motion state vector using a recurrent neural network of a long-short memory (LSTM) method using the first motion state vector as an input. can do. The cyclic neural network may be configured as a deep learning network.

In step 1140, the device 200 generates a current predicted athletic state array based on an athletic model, a previous predicted athletic state array, and an Nth predicted athletic state vector. The athletic model can include a linear athletic model or a non-linear athletic model. An appropriate athletic model may be selected from among the plurality of athletic models based on the generated IMU data.

12 illustrates a method of estimating an attitude of an i + x point of view based on an attitude of an i th view according to an example.

The apparatus 200 may be an attitude estimation unit. If the current view is the i-th view, the determined posture P _i of the i-th view is the input of the posture estimation unit, and the output of the posture estimation unit is the predicted posture P (i. _{i + x} ).

Since the IMU data is iteratively and continuously generated until the i + x time point, the predicted attitude P ″ _{i + x} at the i + x time point is continuously generated through circular detection until the i + x time point is reached. Can be updated.

Detailed operation of the attitude estimation unit is as follows.

i) The difference between the posture P _{i of} the i-th time point determined from the IMU data and the predicted posture P ' _{i of} the i-th time point predicted from the i-th time point is calculated. Error may be used to adjust the movement state vector (S _i) of the i-th point. As needed, an additional Kalman gain (Kg _i) may be used to adjust the movement state vector (S _i).

ii) the movement state vector (S _i) and a movement model (M) the i + 1 position prediction (P _{'i + 1)} of the start point on the basis of the i-th point in time is calculated. The predicted attitude P ′ _{i + 1} at the i + 1th time point is an input of the attitude estimation unit at the i + 1th time point.

iii) i + N-1 at the i + N-1 time point using a recurrent neural network of a Long-Short Term Memory (LSTM) method using the motion state vector S _i as an input; Generate a predicted motion state vector (S ' _{i + N-1} ). Although the embodiment of FIG. 12 uses an exercise state vector, other embodiments may use other information representing the state of the device 200 and are not limited to the disclosed embodiment.

For example, the i + N−1 predicted motion state vector S ′ _{i + N−1} may be calculated using Equation 2 below.

vi) Generates the current predicted exercise state array based on the exercise model M, the previous predicted exercise state array and the i + N-1 predicted exercise state vector S ′ _{i + N-1} .

If the present point is changed to the i + 1 time, the i + prediction of the first point position (P _{'i + 1)} and before the calculated current prediction motion state array is circulated to the posture estimation unit as previously predicted motion state arranged in Can be entered. The above operations may be cyclically performed until the current time point becomes the i + x time point. By using the cyclic scheme, an error between the i + x-th view prediction attitude and the i + x-th view prediction attitude determined using the IMU data may be reduced.

13 illustrates a method of calculating a current predicted exercise state arrangement according to an example.

The i + N-1 predicted motion state vector S ′ _{i + N-1} generated using the LSTM method may be added to the LSTM predicted motion state array. For example, the LSTM predicted motion state arrangement may include N data and add the latest data in a first in first out (FIFO) manner. The LSTM predictive motor state arrangement may be a motion state sliding window.

The current predicted exercise state arrangement may be calculated based on a previous predicted exercise state arrangement (the present predicted exercise state array at the i-1 point in time) and the LSTM predicted exercise state array calculated at the past point in time, i-1. For example, the current predicted athletic state arrangement may be calculated by combining the previous predicted athletic state arrangement and the LSTM predicted athletic state arrangement. The more recent the predicted motion state vector, the greater the weighting may be.

A prediction pose P ″ _{i + x} of the i + x time point may be generated based on the generated current predicted motion state arrangement. For example, the i + x time point may be generated using Equation 2 below. The predicted attitude P " _{i + x of} can be calculated.

14 illustrates a method of detecting a static state according to an example.

The thick line is the variance change curve of the IMU data as the user walks. Due to the repeatability of user walking, the distribution of IMU data changes periodically. If the local variance (variance in the time window) of the IMU data is less than the predicted threshold, it corresponds to the phase where the foot is in contact with the ground in response to the center of mass, which is the zero-speed interval. It is called. In one embodiment, the zero velocity interval is extracted during the exercise of the device 200 using the threshold value, and this interval is added as a constraint to the cluster optimization.

15 illustrates a result due to jitter generated by circular detection according to an example.

In addition to errors in the IMU integration that can cause differences between poses, cyclical detection of the same map may cause differences between poses. 15 shows a scene in which jitter occurs due to cyclic detection.

After the device 200 has taken a turn, when there is a previously routed location, there may be a constant deviation between the previously stored map point and the recently stored map point. At this time, it is necessary to update the position of the current map point by using the circular detection function. That is, it is necessary to correct both the posture difference caused by the error in the IMU integration and the posture difference caused by the circulation.

An inverse relationship is established between the speed of movement of the object projected onto the retina and the sensitivity of sensing the movement of the object with the human eye. Accordingly, by detecting the movement speed H _S of the eyeball and the movement speed P of the virtual object, a speed H _T detected by a person may be calculated. Then, the speed at which the bias decreases is determined using the speed H _T sensed by a person. The larger the human perceived speed H _T , the faster the bias reduction. The speed H _T detected by a person can be calculated using Equation 4 below.

는 대부분 상황에서 0.6이고,

는 오브젝트의 크기이고,

는 망막에 투사되는 오브젝트의 크기이다.In [Equation 4],

Is 0.6 in most situations,

Is the size of the object,

Is the size of the object projected onto the retina.

16 is a flowchart of a method of outputting a virtual object according to an example.

In an AR system or a VR system, after determining the posture of the device 200 at the present time using a Simulaneous Localization and Mapping (SLAM) calculation method, there is a need to manufacture a virtual object according to the posture of the device 200. In addition, the created virtual object can be output or displayed by being superimposed on the real object. When the creation of a virtual object is complicated or lighting calculation is included, it takes a certain time to generate the virtual object. This causes a certain delay in the output of the virtual object, which may affect the AR or VR effect. The following describes an image output method that can reduce this delay.

Steps 1610 to 1650 below may be performed after step 1040 described above with reference to FIG. 10.

In operation 1610, the device 200 generates a virtual object to be output at the Mth time point. For example, the virtual object may be a holographic image or hologram.

The holographic image and the depth image generated for the previous image can be used to create a new holographic image. Since the depth information of each point of the previous image and the attitude of the device 200 corresponding to the hologram image of the two images are known in advance, the position of the pixel of the previous image corresponding to the pixel of the image to be generated may be calculated. have. Copies the pixels of the previous image to the pixels of the image to be created corresponding to the pixels. This method reduces the amount of computation to be processed, thereby reducing the creation time of the virtual object.

In step 1620, the device 200 adjusts the virtual object.

The device 200 obtains a posture change of the device 200 that occurs during the rendering of the virtual object, and adjusts the virtual object to correspond to the acquired posture change.

The time point at which the virtual object is output may be the Mth time point. When the frequency of generating the IMU data is greater than the frequency at which the virtual object is output, the IMU data may be generated at the time points between the time points at which the virtual object is output. For example, the motion state of the device 200 between the first time point and the Mth time point may be obtained by integrating the IMU data. A posture change may be obtained based on the acquired exercise state, and the virtual object may be adjusted using the acquired posture change.

In step 1630, the device 200 outputs the virtual object. For example, the device 200 may generate an image for the Mth view and output a virtual object to synthesize the generated image and the virtual object. As another example, the device 200 may output only a virtual object.

17 illustrates a method of outputting a composite image based on an estimated attitude of the apparatus according to an example.

The attitude of the apparatus 200 from the first time point to the Mth time may be estimated. The virtual object to be output at the Mth view may be generated (or rendered) from the time at which the attitude of the apparatus 200 at the Mth view is estimated.

IMU data may be obtained at points in time between the first time point and the Mth time point, and the attitude of the apparatus 200 at the Mth time point may be continuously estimated based on the obtained IMU data. The virtual object may be adjusted to correspond to the posture of the re-estimated device 200. The virtual object is displayed at the M point of time.

18 is a block diagram illustrating modules of an apparatus according to an example.

The apparatus 200 may include a SLAM module, an attitude estimation module, and an image generation module. The SLAM module utilizes the fusion of a plurality of sensors and generates the current pose of the device 200 at a high frame rate. The attitude estimation module estimates the future attitude of the device 200 based on the output of the SLAM module and the data filtering. The image generation module adjusts the generated virtual object or image according to the change of attitude of the device 200 during the virtual object generation period.

19 is a view illustrating posture estimation results according to various detection methods according to an example.

Among the attitude estimation results according to various detection methods, the method using the IMU integral filtering shows the flattest estimation result. In the case of performing the first cluster optimization step and the second cluster optimization step continuously, by using filtering to integrate IMU data and remove noise, etc., it is possible to prevent the posture of the device 200 from being estimated to have a sudden change. Can be.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the following claims.

200: device
210: communication unit
220: processor
230: memory
240: IMU
250: camera

Claims (17)

In the attitude estimation method of the device, performed by the device,
Generating IMU data using an Inertial Measurement Unit (IMU) of the device;
Determining a first pose of the device based on the IMU data;
Generating a present prediction motional state array based on the IMU data; And
Estimating an Mth prediction pose at an Mth point after a first time point of the first pose based on the current predicted exercise state arrangement;
Including,
M is a natural number greater than 1,
Posture estimation method.
The method of claim 1,
Generating the current prediction exercise state arrangement,
Smoothing the IMU data;
Generating a first motional state vector of the first time point based on the flattened IMU data;
Determining an Nth predicted motion state vector at an Nth time point based on the first motion state vector, wherein N is a natural number greater than M; And
Generating the current predicted kinematic state array based on a motion model, a previous predicted kinematic state array and the Nth predicted kinematic state vector;
Including,
Posture estimation method.
The method of claim 2,
Generating the first exercise state vector may include:
Generating the first motion state vector by correcting the flattened IMU data based on a difference between the first pose and a first estimated pose that is previously estimated.
Including,
Posture estimation method.
The method of claim 2,
The determining of the N th predicted exercise state vector may include:
Generating the N-th predicted motion state vector by using a recurrent neural network of a Long-Short Term Memory (LSTM) method using the first motion state vector as an input;
Including,
Posture estimation method.
The method of claim 1,
Generating the first exercise state vector may include:
Determining a frequency of generating the first exercise state vector based on at least one of an ocular movement velocity of the user of the device and a head movement velocity of the user.
Including,
Posture estimation method.
The method of claim 1,
Determining whether the state of the device is static based on the IMU data
More,
Determining the first posture of the device,
Determining a first posture based on whether the state of the device is a static state
Including,
Posture estimation method.
The method of claim 1,
Generating a virtual object to be output at the Mth time point; And
Outputting the virtual object
Further comprising,
Posture estimation method.
The method of claim 7, wherein
Adjusting the virtual object based on a change in attitude of the device that occurs during the rendering of the virtual object
Further comprising,
Posture estimation method.
The method of claim 7, wherein
The outputting of the virtual object may include:
Outputting a virtual object to synthesize the virtual object and an image for a Mth viewpoint
Including,
Posture estimation method.
The method of claim 7, wherein
The virtual object is a holographic image,
Posture estimation method.
The method of claim 1,
Generating a map for the surrounding space of the device;
Determining an initial pose using the map; And
Determining whether the device is in a static state based on IMU data
More,
The first posture is determined when the device is in a static state,
Posture estimation method.
The method of claim 11,
Generating the map,
Generating a left image using the left camera of the device;
Generating one or more first feature points from the left image;
Generating a right image using the right camera of the device;
Generating one or more second feature points from the right image;
Matching the first feature points and the second feature points; And
Generating the map based on matched and unmatched feature points of the first and second feature points.
Including,
Posture estimation method.
The method of claim 11,
Determining the initial posture,
Generating one or more feature points from an image generated by at least one of the left camera and the right camera;
Determining a target point corresponding to the one or more feature points among candidate feature points preset in the map; And
Determining the initial posture based on the target feature point
Including,
Posture estimation method.
The method of claim 13,
Determining the initial posture,
Determining a remaining feature point of the one or more feature points that does not correspond to candidate feature points preset in the map; And
Updating the map by adding feature points corresponding to the remaining feature points to the map.
Further comprising,
Posture estimation method.
The method of claim 1,
The device,
A wearable device,
Posture estimation method.
A computer-readable recording medium containing a program for performing the method of any one of claims 1 to 15.
The device for estimating a posture,
At least one camera;
Inertial Measurement Unit (IMU);
A memory in which a program for estimating posture is recorded; And
A processor that executes the program
Including,
The program,
Generating IMU data using an Inertial Measurement Unit (IMU) of the device;
Determining a first pose of the device based on the IMU data;
Generating a current predicted workout state arrangement based on the IMU data; And
Estimating an Mth predictive posture at an Mth time after a first time point of the first posture based on the current predicted exercise state arrangement, wherein M is a natural number greater than 1
To do,
Posture estimation method.

Images