KR102442780B1 - Method for estimating pose of device and thereof - Google Patents

Abstract

A method and apparatus for estimating a posture of a device are provided. In order to estimate the posture of the device, IMU data of the device is generated, the posture of the current time is determined based on the IMU data, and the current posture and the current predicted motion state array generated based on the IMU data are used to determine the posture of the future time point. Estimate your posture

Description

Method for estimating posture of a device and apparatus thereof

The following embodiments relate to a method and apparatus for estimating the posture of a device, and more particularly, to a method and apparatus for estimating the posture of a device using an inertial measurement unit (IMU).

Display technologies such as "virtual reality (VR)" or "augmented reality (AR)" require the output image to feel to the user as if it were real. Users can experience various experiences that cannot be experienced in reality through these technologies. Real-time is important for images provided to users through VR or AR. In particular, the output image closely related to the user's posture needs to accurately detect the user's posture. When the real-time performance of the output image is low, the user feels uncomfortable due to an error occurring in the difference between the user's posture and the output image.

For example, a device worn by the user may be used to detect the user's posture. The tight coupling method of detecting the posture of the device through many sensors has an advantage of having high accuracy, but there are many constraints to be considered due to a large amount of data, so the calculation speed is slow and a delay occurs.

One embodiment may provide a method and apparatus for estimating the location of a device.

An embodiment may provide a method and an apparatus for estimating a location of a device using a loose coupling method.

According to one aspect, a method for estimating a posture of a device performed by a device includes generating IMU data using an inertial measurement unit (IMU) of the device, and generating IMU data based on the IMU data. determining a first pose, generating a present prediction motional state array based on the IMU data, and based on the current predicted motional state array, and estimating an M-th prediction pose at the M-th time point after the first time point.

The generating of the current predicted motion state array may include smoothing the IMU data, and generating a first motional state vector of the first time based on the flattened IMU data. Step, determining an Nth predicted motional state vector of an Nth time based on the first motional state vector, and a motional model, a previous prediction motional state array, and the Nth and generating the current predicted motion state array based on the predicted motion state vector.

The generating of the first motion state vector may include generating the first motion state vector by correcting the flattened IMU data based on a difference between the first posture and a pre-estimated first prediction pose. may include the step of

The determining of the N-th predicted motion state vector may include predicting the N-th using a recurrent neural network of a Long-Short Term Memory (LSTM) method using the first motion state vector as an input. generating a motion state vector.

The generating of the first motion state vector may include generating the first motion state vector based on at least one of an ocular movement velocity and a head movement velocity of a user of the device. It may include the step of determining the frequency of

The method for estimating the posture further comprises determining whether the state of the device is in a static state based on the IMU data, wherein determining the first posture of the device includes whether the state of the device is in a static state It may include determining the first posture based on whether or not.

The method of estimating the posture may further include generating a virtual object to be output at the Mth time point, and outputting the virtual object.

The method for estimating the posture may further include adjusting the virtual object based on a change in the posture of the device that occurs during a period of rendering the virtual object.

The outputting of the virtual object may include outputting the virtual object so that the image for the M-th viewpoint and the virtual object are synthesized.

The virtual object may be a holographic image.

The method for estimating the posture includes generating a map for the surrounding space of the device, determining an initial pose using the map, and setting the device to a static state based on IMU data. The method may further include determining whether to recognize the first posture, wherein the first posture may be determined when the device is in a static state.

The generating of the map may include generating a left image using a left camera of the device, generating one or more first feature points from the left image, and generating a right image using a right camera of the device generating one or more second feature points from the right image, matching the first feature points and the second feature points, and matching and unmatched first feature points and among the second feature points It may include generating the map based on the feature points that are not.

The determining of the initial posture may include generating one or more feature points from an image generated by at least one of the left camera and the right camera, and a candidate feature point preset in the map. The method may include determining a target feature point corresponding to the one or more feature points, and determining the initial posture based on the target feature point.

The determining of the initial posture may include determining a remaining feature point that does not correspond to candidate feature points preset in the map among the one or more feature points, and a feature point corresponding to the remaining feature point on the map The method may further include updating the map by adding to .

The device may be a wearable device.

According to another aspect, an apparatus for estimating an attitude includes at least one camera, an inertial measurement unit (IMU), a memory in which a program for estimating an attitude is recorded, and a processor for executing the program, The program may include generating IMU data using an inertial measurement unit (IMU) of the device, determining a first pose of the device based on the IMU data, and the IMU data generating a current predicted motion state array based on .

1 illustrates images from which a virtual object is output according to an example.
2 is a block diagram of an apparatus according to an embodiment.
3 is a flowchart of a cluster optimization method for estimating a posture of a device according to an example.
4 is a flowchart of a first cluster optimization method according to an example.
5 is a flowchart of a method of generating a map for a surrounding space of an apparatus according to an example.
6 is a flowchart of a method of determining an initial posture using a map according to an example.
7 illustrates a method of optimizing a cluster while determining an initial posture according to an example.
8 is a flowchart of a method for updating a map according to an example.
9 illustrates a second cluster optimization method according to an example.
10 is a flowchart of a method of estimating an M-th predicted posture through second cluster optimization according to an embodiment.
11 is a flowchart of a method for generating a current predicted motion state arrangement according to an example.
12 illustrates a method of estimating a posture at an i+x time point based on the posture at the i-th time point according to an example.
13 illustrates a method of calculating a current predicted motion state arrangement according to an example.
14 illustrates a method for detecting a static state according to an example.
15 illustrates a result due to jitter generated by cyclic detection according to an example.
16 is a flowchart of a method of outputting a virtual object according to an example.
17 illustrates a method of outputting a composite image based on an estimated posture of an apparatus according to an example.
18 is a block diagram of modules of an apparatus according to an example.
19 illustrates posture estimation results according to various detection methods according to an example.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference numerals in each figure indicate like elements.

Various modifications may be made to the embodiments described below. It should be understood that the embodiments described below are not intended to limit the embodiments, and include all modifications, equivalents, and substitutions thereto.

Terms used in the examples are only used to describe specific examples, and are not intended to limit the examples. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 illustrates images from which a virtual object is output according to an example.

For example, the first image 110 includes a real object 111 and a virtual object 112 . The virtual object 112 may be additional information about the real object 111 . The position of the virtual object 112 in the first image 11 may be determined based on the user's gaze and the position of the real object 111 .

When the user's gaze changes, the second image 120 changes. The second image 120 includes a real object 121 corresponding to the real object 110 and a virtual object 122 corresponding to the virtual object 112 . The position of the real object 121 in the second image 120 is immediately changed by reflecting the user's gaze. However, when the user's gaze is not immediately detected, the position of the virtual object 122 in the second image 120 is not changed and may be determined to be the same as the position of the virtual object 112 . Accordingly, the relative positions between the real object 121 and the virtual object 122 appearing in the second image 120 may not be appropriate.

According to an aspect, the gaze of the user may be detected by determining a posture of the device corresponding to the gaze of the user. The pose of the device may be determined or estimated via sensors of the device. Hereinafter, a method of estimating the posture of the device will be described in detail with reference to FIGS. 2 to 18 .

Hereinafter, the term “posture” refers to a current posture determined using currently measured data, and the term “predicted posture” refers to a future posture estimated using the currently measured data.

2 is a block diagram of an apparatus according to an embodiment.

The device 200 includes a communication unit 210 , a processor 220 , a memory 230 , a camera 240 , and an inertial measurement unit (IMU). The device 200 may be an electronic device. For example, the device 200 may be a device that outputs an image for providing AR or VR to the user. According to one aspect, the device 200 may be a wearable device. For example, the device 200 may be in the form of glasses or a head mounted display (HMD), but is not limited thereto. According to another aspect, the device 200 may be used for measuring a location of a user, estimating a location of a user, and the like. According to another aspect, the device 200 may be used for autonomous driving.

The communication unit 210 is connected to the processor 220 , the memory 230 , the camera 240 , and the IMU 250 to transmit and receive data. The communication unit 210 may be connected to another external device to transmit/receive data.

The communication unit 210 may be implemented as circuitry in the device 200 . For example, the communication unit 210 may include an internal bus and an external bus. As another example, the communication unit 210 may be an element that connects the device 200 and an external device. The communication unit 210 may be an interface. The communication unit 210 may receive data from an external device and transmit the data to the processor 220 and the memory 230 .

The processor 220 processes data received by the communication unit 210 and data stored in the memory 230 . A “processor” may be a data processing device implemented in hardware having circuitry having a physical structure for performing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

Processor 220 executes computer readable code (eg, software) stored in memory (eg, memory 230 ) and instructions issued by processor 220 .

The memory 230 stores data received by the communication unit 210 and data processed by the processor 220 . For example, the memory 230 may store a program. The stored program may be a set of syntaxes coded to estimate the posture of the device and executable by the processor 220 .

According to one aspect, memory 230 may include one or more volatile memory, non-volatile memory and random access memory (RAM), flash memory, hard disk drive, and optical disk drive.

The memory 230 stores a set of instructions (eg, software) for operating the device 200 . The set of instructions for operating the device 200 is executed by the processor 220 .

The camera 240 creates an image by photographing the scene. For example, the camera 240 may include a left camera that generates a left image to be output to the user's left eye and a right camera that generates a right image to be output to the user's right eye. There is a binocular disparity between the left and right images. As another example, the camera 240 may generate one image, and the processor 220 may generate a left image and a right image using an image processing method such as image warping.

The IMU 250 may generate IMU data by measuring an acceleration and an angular velocity according to the movement of the device 200 .

Although not shown, the device 200 may include sensors capable of measuring the position or posture of the device 200 . For example, device 200 may include a global positioning system (GPS) and is not limited to the described embodiments.

The communication unit 210 , the processor 220 , the memory 230 , the camera 240 , and the IMU 250 will be described in detail below with reference to FIGS. 3 to 18 .

3 is a flowchart of a cluster optimization method for estimating a posture of a device according to an example.

According to an aspect, a first cluster optimization step 310 and a second cluster optimization step 320 may be performed to estimate the posture of the device 200 . Cluster optimization refers to optimizing parameters and algorithms used to estimate the posture of the device 200 .

The first cluster optimization step 310 may be performed using a tight coupling cluster optimization method. The tight coupling cluster optimization method estimates the posture of the device 200 using data measured using a large number of sensors. For example, an image generated using the camera 240 and IMU data generated using the IMU 250 may be used to estimate the posture of the device 200 . The first cluster optimization step 310 is described in detail below with reference to FIGS. 4 to 8 .

The second cluster optimization step 320 may be performed using a loose coupling cluster optimization method. The loose coupling cluster optimization method estimates the posture of the device 200 using data measured using a small number of sensors. For example, only IMU data generated using the IMU 250 may be used to estimate the posture of the device 200 . Since the IMU data includes the acceleration and angular velocity of the device 200 , the current posture may be determined by integrating the IMU data and superimposing the integrated value on the previous posture.

The less data used to estimate the posture, the faster the posture can be estimated. The second cluster optimization step 320 is described in detail below with reference to FIGS. 9 to 15 .

4 is a flowchart of a first cluster optimization method according to an example.

Step 310 described above with reference to FIG. 3 includes the following steps 410 to 440 .

In step 410 , the device 200 creates a map of the surrounding space of the device 200 . A map may be generated using images captured by the camera 240 of the device 200 . A method of generating a map is described in detail below with reference to FIG. 5 .

In step 420 , the device 200 determines an initial pose of the device 200 using the map. A method of determining the initial posture will be described in detail below with reference to FIG. 6 .

In step 430 , the device 200 determines whether the state of the device 200 is a static condition based on the IMU data. The static state means that there is no movement of the device 200 or that movement below a preset threshold value appears.

IMU data typically includes noise, and the noise causes an error in the guess posture of the device 200 . When the posture is estimated using the static state (or the still state) of the apparatus 200 , the influence of noise may be reduced. Whether the device 200 is in a static state can be detected by analyzing local variations in the IMU data. When the device 200 is in a stationary state, a constraint is added to increase the accuracy of the posture estimation when performing cluster optimization at the next time point. The constraints are as follows.

i) The location of the device 200 may be obtained based on a spatial location (eg, a three-dimensional location) of a map point and a location of a feature point of an image (left image/or right image) corresponding thereto. ii) The location of the new map point in space may be calculated by the location of the device 200 and the location of the feature point corresponding to the new map point on the image. iii) The position and posture of the device between the before and after images may be calculated through IMU data and IMU parameters between the before and after images. The IMU parameters may be motion state vectors. iv) IMU parameters may be calculated using IMU data between the plurality of images and the posture of the apparatus 200 corresponding to the plurality of images. v) Through the detected still state, it may be determined that the posture between the two images changes to zero.

Since there is a constraint between the map point, the posture of the device 200, and the IMU data, the posture of the device 200 that satisfies the constraint as much as possible may be determined. The more the number of the above constraints is satisfied, the more accurate the posture can be estimated, but the estimation speed becomes slower.

In step 440 , it is determined whether the first cluster optimization is to be performed again or the second cluster optimization is to be performed according to the state of the device 200 .

Through the first cluster optimization step 310 , parameters and algorithms to be used for posture estimation in the second cluster optimization step 320 may be optimized. Parameters and algorithms to be used for posture estimation in the second cluster optimization step 320 will be described in detail below with reference to FIGS. 10 to 13 .

5 is a flowchart of a method of generating a map for a surrounding space of an apparatus according to an example.

Step 410 described above with reference to FIG. 4 includes the following steps 510 to 540 .

In step 510 , the device 200 may generate a left image. For example, the left image is generated using the left camera. As another example, the device 200 may generate the left image by warping the reference image captured using the camera.

In step 515 , the device 200 generates one or more first feature points in the generated left image. The feature point may be an edge and a texture in an image. For example, the device 200 may use methods such as brisk and sift to generate the feature points.

In step 520 , the device 200 may generate the right image. For example, the right image is generated using the right camera. As another example, the device 200 may generate the right image by warping the reference image captured using the camera.

In step 525 , the device 200 generates one or more second feature points in the generated right image.

In step 530 , the device 200 matches the first feature point and the second feature point. For example, the apparatus 200 may verify a matching relationship using a fitting via random sample consensus (RANSAC) method, and may remove an erroneous match.

In operation 540 , the apparatus 200 generates a map based on matched and unmatched key points among the first and second key points.

6 is a flowchart of a method of determining an initial posture using a map according to an example.

Step 420 described above with reference to FIG. 4 includes the following steps 610 to 630 .

In step 610 , the device 200 generates one or more feature points from the image generated by the camera.

In operation 620 , the apparatus 200 determines a target feature point corresponding to one or more feature points among candidate feature points preset in the map.

In step 630 , the device 200 determines an initial posture of the device 200 based on the target feature point. For example, the initial posture may be a current posture of the camera 240 (or the device 200 ) corresponding to the user's gaze. The posture may mean a view point of the camera.

While step 310 is iteratively performed, parameters for estimating a future posture of the device 200 may be optimized. The parameters optimized through step 310 may be used continuously in step 320 .

7 illustrates a method of optimizing a cluster while determining an initial posture according to an example.

The first posture 710 of the device 200 is determined using the image and the map captured in the first posture 710 . For example, target feature points 721 and 722 of the map may be determined based on the image captured in the first posture 710 . A user may wear the device 200 and move. While moving, the motion state of the device 200 may be detected. The motion state may be IMU data. Based on the determined first posture 710 and the IMU data, the posture of the device 200 after the first posture 710 may be estimated. In order to reduce the error of the estimated posture, noise-filtered IMU data may be used. For example, noise biased IMU data may be used to estimate pose.

The second posture 711 of the device 200 is determined by using the image and the map captured in the second posture 711 . For example, target feature points 721 , 722 , 723 , 724 , and 725 of the map may be determined based on the image captured in the second posture 711 . Based on the determined second posture 711 and the IMU data, the posture of the device 200 after the second posture 711 may be estimated.

A third posture 712 of the device 200 is determined using the image and the map captured at the third posture 712 . For example, target feature points 723 , 725 , 726 , 728 , and 729 of the map may be determined based on the image captured in the third posture 712 . Based on the determined third posture 712 and the IMU data, the posture of the device 200 after the third posture 712 may be estimated.

A fourth posture 713 of the device 200 is determined using the image and the map captured at the fourth posture 713 . For example, target feature points 726 , 727 , 728 , and 729 of the map may be determined based on the image captured in the fourth posture 713 . The posture of the device 200 after the fourth posture 713 may be estimated based on the determined fourth posture 712 and the IMU data.

The first posture 710 to the fourth posture 713 may be determined as the actual posture, and parameters for estimating the posture based on the actual posture may be adjusted. Through iterative estimation, the error between the actual posture and the estimated posture can be reduced.

Each of the first postures 710 to 713 may be an initial posture, and it may be understood that the initial posture is updated for optimizing the second cluster.

8 is a flowchart of a method for updating a map according to an example.

Step 420 described above with reference to FIG. 4 may further include steps 810 and 820 below.

In operation 810 , the apparatus 200 determines remaining feature points that do not correspond to candidate feature points preset in the map among one or more feature points of the generated image.

In step 810 , the device 200 updates the map by adding the feature points corresponding to the remaining feature points to the map. The added feature point becomes a candidate feature point. As the candidate feature points increase, the accuracy of the posture of the device 200 determined using the feature points may increase.

9 illustrates a second cluster optimization method according to an example.

The second cluster optimization method using the loose coupling cluster optimization method is a method of rapidly estimating the posture of the apparatus 200 using data measured by a small number of sensors. For example, a smaller number of data than data used in the first cluster optimization method may be used. Since a small number of data is used, the posture estimation speed of the apparatus 200 may be increased.

According to an aspect, a current posture of the device 200 may be determined using IMU data, and a future posture of the device 200 may be estimated using a history of the IMU data. Cyclic detection can be used to reduce the difference between the current pose and the estimated pose.

Hereinafter, a method of estimating the posture of the apparatus 200 using the loose coupling cluster optimization method will be described in detail with reference to FIGS. 10 to 15 .

10 is a flowchart of a method of estimating an M-th predicted posture through second cluster optimization according to an embodiment.

Step 320 described above with reference to FIG. 3 may include the following steps 1010 to 1040 . According to one aspect, step 320 may be performed when the state of the device 200 is a static state.

At step 1010 , device 200 generates IMU data using IMU 250 .

In step 1020 , the device 200 determines a first pose of the device 200 based on the IMU data. The first posture is the posture of the device 200 at the current time point (time i).

According to an aspect, the device 200 may determine the first posture based on whether the state of the device 200 is a static state. For example, when the state of the apparatus 200 is a static state, the device 200 determines the first posture by using the IMU data.

In step 1030 , the device 200 generates an array of current predicted motion states based on the IMU data. The term “predicted motion state arrangement” means that vectors for predicting motion states of the device 200 are arranged according to time points. The term “present prediction motional state array” refers to a predicted motional state array estimated at a current time point (time i). The term "previous prediction motional state array" means an array of predicted motional states estimated at a previous time point (time i-1). That is, the current predicted motion state sequence may be generated by updating the previous predicted motion state sequence.

A method of generating the current predicted motion state arrangement is described in detail below with reference to FIGS. 11 to 13 .

In step 1040 , the device 200 estimates the Mth predicted posture at the Mth time point after the first posture based on the current predicted motion state arrangement. M is a natural number greater than 1.

11 is a flowchart of a method for generating a current predicted motion state arrangement according to an example.

Step 1030 described above with reference to FIG. 10 includes the following steps 1110 to 1140 .

In step 1110 , the device 200 smoothes the measured IMU data to remove noise and outliers. For example, a Kalman filter may be used to flatten the IMU data.

In step 1120 , the device 200 generates a first motion state vector of a first time point based on the flattened IMU data. For example, the motion state vector may be expressed by [Equation 1] below.

In [Equation 1], S is a motion state vector, v is a motion speed, a _v is a motion acceleration, w is a turning speed, and a _w is a turning acceleration. The turning speed and turning acceleration can be added to indicate the respective six-axis directions. In addition, the motion state vector may further include a position of the camera 240 .

According to one aspect, the device 200 may generate the first motion state vector by correcting the flattened IMU data based on a difference between a first pose and a pre-estimated first prediction pose. . A difference between the first posture and the pre-estimated first predicted posture may be defined as a replenishment amount.

According to one aspect, the device 200 is configured to generate a first motion state vector based on at least one of a user's ocular movement velocity and a user's head movement velocity of the device 200 . frequency can be determined. For example, when compensating for a difference caused by acclimatization detection, the rate of replenishment is related to the rate of eye movement of the user and/or rate of movement of the user's head. The faster the head movement speed, the faster the replenishment rate.

In operation 1130 , the device 200 determines an N-th motion state vector at an N-th time point based on the first motion state vector. For example, the N-th view may be a view of the farthest point predictable from the current view. N is a natural number greater than M.

According to one aspect, the apparatus 200 generates an N-th predicted motion state vector using a recurrent neural network of a Long-Short Term Memory (LSTM) method using the first motion state vector as an input. can do. The recurrent neural network may be configured as a deep learning network.

In step 1140 , the device 200 generates a current predicted motion state sequence based on the motional model, the previous predicted motion state sequence, and the Nth predicted motion state vector. The motion model may include a linear motion model or a non-linear motion model. An appropriate motion model may be selected from among a plurality of motion models based on the generated IMU data.

12 illustrates a method of estimating a posture at an i+x time point based on the posture at the i-th time point according to an example.

The device 200 may be a posture estimation unit. If the current time point is the i-th time point, the determined posture P _i at the i-th time point is an input of the posture estimation unit, and the output of the posture estimation unit is the predicted posture P " at the i+x time point using the predicted motion state arrangement. _i+x ).

Since the IMU data is repeatedly and continuously generated until the i+x time point, the predicted posture (P" _i+x ) of the i+x time point is continuously maintained through the cyclic detection until the i+x time point is reached. may be updated.

A detailed operation process of the posture estimation unit is as follows.

i) The difference between the posture (P _i ) of the i-th time determined through the IMU data and the predicted posture (P′ _i ) of the i-th time predicted at the i-1 time is calculated. The error may be used to adjust the motion state vector (S _i ) of the i-th time point. The Kalman gain (Kg _i ) may be used to further adjust the motion state vector (S _i ) as needed.

ii) The predicted posture (P′ _i+1 ) at the i+1th time point is calculated based on the motion state vector (S _i ) and the motion model (M) at the i-th time point. The predicted posture P′ _i+1 at the i+1th time point becomes an input to the posture estimation unit at the i+1th time point.

iii) Using a recurrent neural network of a Long-Short Term Memory (LSTM) method with a motion state vector (S _i ) as an input, i+N-1 at the i+N-1 time point A predicted motion state vector (S' _i+N-1 ) is generated. In the embodiment of FIG. 12 , a motion state vector is used, but in another embodiment, other information indicating the state of the device 200 may be used, and the embodiment is not limited thereto.

For example, the i+N-1th predicted motion state vector (S′ _i+N-1 ) may be calculated using Equation 2 below.

vi) generate the current predicted motion state sequence based on the motion model M, the previous predicted motion state sequence, and the i+N-1 predicted motion state vector (S′ _i+N-1 ).

When the current time point changes to the i+1th time point, the predicted posture (P′ _i+1 ) of the i+1th time point and the previously calculated current predicted motion state arrangement are circulated to the posture estimation unit as the previous predicted motion state arrangement. can be entered. The above operations may be cyclically performed until the current time point becomes the i+x time point. An error between the predicted posture at the i+x time point and the posture predicted using the IMU data may be reduced by using the cycle method.

13 illustrates a method of calculating a current predicted motion state arrangement according to an example.

The i+N-1th predicted motion state vector (S′ _i+N-1 ) generated using the LSTM method may be added to the LSTM predicted motion state array. For example, the LSTM prediction motion state array may include N pieces of data, and the latest data may be added in a first in first out (FIFO) manner. The LSTM prediction motion state arrangement may be a motional state sliding window.

The current predicted motion state arrangement may be calculated based on the previous predicted motion state arrangement (the current predicted motion state arrangement at the i-1th time point) and the LSTM predicted motion state arrangement calculated at the i-1 th time point, which is a past time point. For example, the current predicted motion state sequence may be calculated by merging the previous predicted motion state sequence and the LSTM predicted motion status sequence. The more recent the predicted motion state vector, the greater the weight may be given.

The predicted posture (P" _i+x ) of the i+x time point may be generated based on the generated current predicted motion state arrangement. For example, the i+x time point using the following [Equation 2] A predicted posture (P" _i+x ) of can be calculated.

14 illustrates a method for detecting a static state according to an example.

The thick line is the variance change curve of the IMU data when the user walks. Due to the repeatability of the user's gait, the distribution of the IMU data changes periodically. If the local variance (dispersion within the time window) of the IMU data is smaller than the predicted threshold, it corresponds to the phase in which the foot touches the ground in response to the center of mass, and this phase corresponds to the zero-speed interval. It is said In one embodiment, a zero velocity section is extracted from the motion process of the device 200 using a threshold value, and this section is added as a constraint to cluster optimization.

15 illustrates a result due to jitter generated by cyclic detection according to an example.

In addition to errors in the IMU integration that can cause differences between postures, differences between postures can also occur due to circular detection of the same map. Fig. 15 shows a scene in which jitter occurs due to cyclic detection.

When the device 200 moves once and reaches a previously passed position, there may be a certain deviation between the previously stored map point and the recently stored map point. In this case, there is a need to update the position of the current map point by using the circular detection function. That is, there is a need to correct both the posture difference induced by the presence of an error in the IMU integration and the posture difference induced by circulation.

An inverse relationship is established between the speed of the object's movement projected on the retina and the sensitivity of the human eye to detect the object's movement. Accordingly, by detecting the movement speed H _S of the eye and the movement speed P of the virtual object, the speed H _T sensed by the person may be calculated. Then, the speed at which the bias decreases is determined using the speed H _T sensed by the person. The larger the human-sensed speed H _T is, the faster the bias reduction is. The speed H _T sensed by a person may be calculated using the following [Equation 4].

는 대부분 상황에서 0.6이고,

는 오브젝트의 크기이고,

는 망막에 투사되는 오브젝트의 크기이다.In [Equation 4],

is 0.6 in most situations,

is the size of the object,

is the size of the object projected onto the retina.

16 is a flowchart of a method of outputting a virtual object according to an example.

In an AR system or a VR system, there is a need to determine the posture of the device 200 at the current viewpoint by using a Simultaneous Localization and Mapping (SLAM) calculation method, and then create a virtual object according to the posture of the device 200 . In addition, the created virtual object can be output or displayed by superimposing it on the real object. When the creation of the virtual object is complicated or lighting calculation is included, it takes a certain amount of time to create the virtual object. This may cause a certain delay in the output of the virtual object, which may affect the AR or VR effect. An image output method that can reduce this delay is described below.

The following steps 1610 to 1650 may be performed after step 1040 described above with reference to FIG. 10 is performed.

In operation 1610 , the device 200 generates a virtual object to be output at the M th time. For example, the virtual object may be a holographic image or a hologram.

The holographic image and depth image created for the previous image can be used to create a new holographic image. Since the depth information for each point of the previous image and the posture of the device 200 corresponding to the holographic image of the two images are known in advance, the position of the pixel of the previous image corresponding to the pixel of the image to be generated can be calculated. have. The pixels of the previous image are copied to the pixels of the image to be created corresponding to those pixels. Since this method reduces the amount of computation to be processed, it is possible to reduce the creation time of the virtual object.

In step 1620 , the device 200 adjusts the virtual object.

The device 200 obtains a posture change of the device 200 that occurs during a rendering period of the virtual object, and adjusts the virtual object to correspond to the obtained posture change.

The time point at which the virtual object is output may be the Mth time point. When the frequency of generating the IMU data is greater than the frequency at which the virtual object is output, IMU data may be generated at time points between the time points at which the virtual object is output. For example, the motion state of the apparatus 200 between the first time point and the M-th time point may be obtained by integrating the IMU data. A posture change may be obtained based on the obtained motion state, and the virtual object may be adjusted using the obtained posture change.

In step 1630 , the device 200 outputs a virtual object. For example, the apparatus 200 may generate an image for the M-th viewpoint, and output the virtual object so that the generated image and the virtual object are synthesized. As another example, the device 200 may output only a virtual object.

17 illustrates a method of outputting a composite image based on an estimated posture of an apparatus according to an example.

The posture of the device 200 from the first time point to the Mth time point may be estimated. A virtual object to be output at the M-th time may be generated (or rendered) from the time when the posture of the apparatus 200 at the M-th time is estimated.

IMU data may be acquired at time points between the first time point and the M-th time point, and the posture of the device 200 at the M-th time point may be continuously re-estimated based on the obtained IMU data. The virtual object may be adjusted to correspond to the re-estimated posture of the device 200 . The virtual object is displayed at the Mth time point.

18 is a configuration diagram of modules of an apparatus according to an example.

The apparatus 200 may include a SLAM module, a posture estimation module, and an image generation module. The SLAM module uses the fusion of multiple sensors and generates the current pose of the device 200 at a high frame rate. The posture estimation module estimates the future posture of the device 200 based on the output and data filtering of the SLAM module. The image generating module adjusts the generated virtual object or image according to a change in the posture of the device 200 during the virtual object creation period.

19 illustrates posture estimation results according to various detection methods according to an example.

Among the posture estimation results according to various detection methods, the method using IMU integral filtering shows the flattest estimation result. When the first cluster optimization step and the second cluster optimization step are successively performed, by integrating the IMU data and using filtering to remove noise, etc., it is possible to prevent the posture of the device 200 from being estimated to have a sudden change. can

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

200: device
210: communication unit
220: processor
230: memory
240: IMU
250: camera

Claims (17)

A method for estimating a posture of a device, performed by the device, comprising:
generating IMU data using an inertial measurement unit (IMU) of the device;
determining a first pose of the device based on the IMU data;
generating a present prediction motional state array based on the IMU data; and
estimating an M th prediction pose at an M th time point after a first time point of the first pose, based on the current predicted motion state arrangement, wherein M is greater than 1 natural Commitment -
including,
The step of generating the current predicted motion state array comprises:
smoothing the IMU data;
generating a first motional state vector of the first time point based on the flattened IMU data;
determining an N-th predicted motion-state vector at an N-th time based on the first motion-state vector, wherein N is a natural number greater than M; and
generating the current predicted motional state array based on a motional model, a previous prediction motional state array, and the Nth predicted motional state vector;
containing,
Posture estimation method.
delete According to claim 1,
The step of generating the first motion state vector comprises:
generating the first motion state vector by correcting the flattened IMU data based on a difference between the first pose and a pre-estimated first prediction pose;
containing,
Posture estimation method.
According to claim 1,
The determining of the N-th predicted motion state vector comprises:
Generating the N-th predicted motion state vector using a recurrent neural network of a long-short-term memory (LSTM) method using the first motion state vector as an input
containing,
Posture estimation method.
According to claim 1,
The step of generating the first motion state vector comprises:
determining a frequency of generating the first motion state vector based on at least one of a user's ocular movement velocity and a user's head movement velocity of the device;
containing,
Posture estimation method.
According to claim 1,
determining whether the state of the device is a static state based on the IMU data;
further comprising,
Determining the first posture of the device comprises:
determining a first posture based on whether the state of the device is a static state;
containing,
Posture estimation method.
According to claim 1,
generating a virtual object to be output at the Mth time point; and
outputting the virtual object
further comprising,
Posture estimation method.
8. The method of claim 7,
adjusting the virtual object based on a change in the posture of the device that occurs during the rendering period of the virtual object;
further comprising,
Posture estimation method.
8. The method of claim 7,
The step of outputting the virtual object comprises:
Outputting a virtual object so that the image for the Mth viewpoint and the virtual object are synthesized
containing,
Posture estimation method.
8. The method of claim 7,
The virtual object is a holographic image (holographic image),
Posture estimation method.
According to claim 1,
generating a map of the surrounding space of the device;
determining an initial pose using the map; and
determining whether the device is in a static state based on IMU data;
further comprising,
wherein the first posture is determined when the device is in a static state;
Posture estimation method.
12. The method of claim 11,
Creating the map includes:
generating a left image using a left camera of the device;
generating one or more first feature points from the left image;
generating a right image using the right camera of the device;
generating one or more second feature points from the right image;
matching the first feature points and the second feature points; and
generating the map based on matched and unmatched feature points among the first feature points and the second feature points
containing,
Posture estimation method.
13. The method of claim 12,
The step of determining the initial posture comprises:
generating one or more feature points from an image generated by at least one of the left camera and the right camera;
determining a target feature point corresponding to the one or more feature points among candidate feature points preset in the map; and
determining the initial posture based on the target feature point
containing,
Posture estimation method.
14. The method of claim 13,
The step of determining the initial posture comprises:
determining, among the one or more feature points, remaining feature points that do not correspond to candidate feature points preset in the map; and
updating the map by adding feature points corresponding to the remaining feature points to the map;
further comprising,
Posture estimation method.
According to claim 1,
The device is
A wearable device,
Posture estimation method.
A computer-readable recording medium storing a program for performing the method of any one of claims 1 to 15.
In the posture estimation device,
at least one camera;
Inertial Measurement Unit (IMU);
a memory in which a program for estimating posture is recorded; and
a processor that executes the program
including,
The program is
generating IMU data using an inertial measurement unit (IMU) of the device;
determining a first pose of the device based on the IMU data;
generating a current predicted motion state array based on the IMU data; and
estimating an M-th predicted posture at an M-th time point after the first time point of the first posture based on the current predicted motion state arrangement, wherein M is a natural number greater than 1;
do,
The step of generating the current predicted motion state array comprises:
smoothing the IMU data;
generating a first motional state vector of the first time point based on the flattened IMU data;
determining an N-th predicted motion-state vector at an N-th time based on the first motion-state vector, wherein N is a natural number greater than M; and
generating the current predicted motional state array based on a motional model, a previous prediction motional state array, and the Nth predicted motional state vector;
containing,
Posture estimation device.

Images