KR20210050997A - Method and apparatus for estimating pose, computer-readable storage medium and computer program for controlling the holder device - Google Patents

Abstract

According to an embodiment of the present invention, a method for estimating a pose comprises the steps of: collecting a plurality of first images from a camera; collecting a plurality of second images from a DB using GPS information on a terminal; estimating a first pose of the camera by comparing any one of the first images with the second images; estimating a second pose of the camera using the first images; and estimating a third pose of the camera by comparing the first and second poses estimated at the same time point. According to an embodiment of the present invention, a more accurate camera pose can be estimated by removing both cases that the first or second pose is incorrect, by using the first and second poses at the same time point.

Description

Pose estimation method and apparatus, computer-readable recording medium and computer program {METHOD AND APPARATUS FOR ESTIMATING POSE, COMPUTER-READABLE STORAGE MEDIUM AND COMPUTER PROGRAM FOR CONTROLLING THE HOLDER DEVICE}

The embodiment relates to a pose estimation method for effectively estimating a pose of a camera provided in a terminal.

Augmented Reality (AR) uses location and direction information to determine an approximate location, and compares facility information such as nearby building information and live-action image information input according to the movement of the camera to provide the service desired by the user. It is a technology that identifies and provides relevant information.

More specifically, augmented reality is a field of virtual reality (VR), which is a computer graphic technique that synthesizes virtual objects into a real environment to make them look like objects existing in the original environment. Unlike the existing virtual reality that targets only objects, it is a technology that can reinforce and provide additional information that is difficult to obtain from the real world by synthesizing virtual objects on the basis of the real world.

With the commercialization of 5G communication, such augmented reality technology is in the spotlight in the field of mobile AR technology used in communication terminals, and current mobile AR technology applications include marker-based mobile AR technology or sensor-based mobile AR technology. have.

The marker-based mobile AR technology is a technology that recognizes the real object through recognition of the marker by photographing a marker corresponding to the real object when photographing a real object to be augmented using a virtual object. 'S mobile AR technology infers the current location of the terminal and the direction you are looking at using the GPS and digital compass installed on the terminal, and overlays POI (Point of Interests) information corresponding to the image in the inferred direction. It is a technology that makes (Overlay).

However, there is a problem that the marker-based mobile AR technology cannot augment a virtual object without a marker, and the sensor-based mobile AR technology accurately displays a virtual object on a specific object due to an error in the current position and direction of the detected terminal. There is a problem that can not be enhanced.

In order to solve the above-described problem, an object of the embodiment is to provide a pose estimation method and a pose estimation apparatus for accurately estimating a pose of a camera.

The pose estimation method of the embodiment includes the steps of collecting a plurality of first images from the camera, collecting a plurality of second images from a DB using GPS information of the terminal, and any one of the plurality of first images. Estimating a first pose of the camera by comparing the first image of the plurality of second images and estimating a second pose of the camera using the plurality of first images, and the same viewpoint And estimating a third pose of the camera by comparing the estimated first pose and the second pose.

In the step of estimating the third pose of the camera, the third pose may use a Mahalanobis distance with respect to a distance between the first pose and the second pose.

In the step of estimating the third pose of the camera, if the Maharanobis distance is less than a first threshold value, the third pose may be estimated using an average value according to a weight of the first pose and the second pose. have.

In the step of estimating the third pose of the camera, if the Mahalanobis distance is greater than a second threshold value greater than the first threshold value, the second pose may be used to estimate the third pose.

In the step of estimating the third pose of the camera, if the Mahalanobis distance is a value between the first threshold and the second threshold, the third pose may be estimated by increasing a weight to the second pose.

In the step of estimating the third pose of the camera, the first pose is measured every first cycle, the second pose is measured every second cycle faster than the first cycle, and the third pose is measured by the first pose. At a point in time that does not occur, the second pose may be estimated as the third pose.

After the step of estimating the third pose, determining a position to implement a virtual object on the screen of the terminal using the third pose.

A pose estimation apparatus according to an embodiment includes a camera image collection unit that collects a plurality of first images captured from a camera of a terminal, and a DB image collection unit that collects a plurality of second images from a DB using GPS information of the terminal. And, a first pose estimating unit that estimates a first pose of the camera by comparing the first image and a plurality of second images, and a second pose estimating a second pose of the camera using the plurality of first images. A second pose estimating unit may include a third pose estimating unit for estimating a third pose of the camera by comparing the first pose and the second pose at the same viewpoint.

According to the embodiment, by using the first pose and the second pose at the same point in time, all cases in which the first pose is inaccurate or the second pose is inaccurate are eliminated, thereby estimating a more accurate camera pose.

1 is a block diagram showing an augmented reality system according to an embodiment.
2 is a block diagram illustrating a pose estimation apparatus according to an embodiment.
3 is a block diagram illustrating a pose estimation method according to an embodiment.
4 is a diagram for explaining a process of collecting a DB image according to an embodiment.
5 is a diagram for describing a step of estimating a third pose according to an exemplary embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings.

1 is a block diagram illustrating an augmented reality system according to an exemplary embodiment, and FIG. 2 is a block diagram illustrating a pose estimation apparatus according to an exemplary embodiment.

Referring to FIG. 1, an augmented reality system 1000 according to an embodiment may include a pose estimation apparatus 100.

The pose estimation apparatus 100 may perform a task of estimating the pose of the camera 240. The pose estimating apparatus 100 includes a processor, and the processor may control the overall operation of the pose estimating apparatus 100.

The pose estimation apparatus 100 may include a transceiver (not shown). The pose estimation apparatus 100 may receive an image from the map generation server 300 using a transceiver and transmit pose information to the terminal 200. The pose may include information on the location and direction of the camera 240.

The pose estimating device 100 uses a transceiver to initialize values from the terminal 200, map information, GPS information of the terminal 200, a 2D image captured by the terminal 200, and/or a camera included in the terminal 200. Information about 240 may be received.

The pose estimation apparatus 100 may include a memory (not shown). The memory may store information required for execution of a program and received images.

The augmented reality system according to the embodiment may include a terminal 200.

The terminal 200 may include a processor 210, a transceiver 220, a memory 230, a camera 240 and a sensor unit 250, and an AR program 260.

The processor 210 may overall control the operation of the terminal 200.

The processor 210 may collect an image captured using the camera 240 and store it as a 2D image.

In addition, the processor 210 further transmits an initialization value, map information of a region to which a virtual object is to be augmented, GPS information, and/or information about the camera 240 to the pose estimating device 100 using the transceiver 220. Can be transmitted.

The processor 210 may receive pose information of the camera 240 generated by the pose estimation apparatus 100 using the transceiver 220.

The processor 210 may execute an augmented reality (AR) program 260. The processor 210 executes the augmented reality program 260, determines a pose of the camera 240 according to the pose value estimated from the pose estimation apparatus 100, and determines the pose of the camera 240 within the image being captured by the camera 240. A position to augment the virtual object may be determined, and the virtual object may be augmented at the determined position.

The processor 210 determines the pose of the camera 240 using the pose information of the camera 240 received from the pose estimation apparatus 100, and updates the pose of the camera 240 using the sensor unit 250 can do.

The memory 230 may store the augmented reality program 250, and the processor 210 may load the augmented reality program 250 and information necessary for execution of the augmented reality program 250 from the memory 230.

The sensor unit 250 may be used to determine the position and direction of the camera 240. The sensor unit 250 may include at least one of a GPS sensor, a tilt sensor, a geomagnetic sensor, a gravity sensor, a gyro sensor, and an acceleration sensor.

The augmented reality system according to the embodiment may include a map generation server 300 (hereinafter referred to as “DB”).

The DB 300 may generate a plurality of second images photographed at various viewpoints of a specific area, and transmit the generated second image to the pose estimation apparatus 100.

The second image is a 2D image, a position (u, v) of each of a plurality of features or feature points included in the 2D image on a 2D image, and an absolute 3D coordinate on the earth of each of the plurality of feature points ( x, y, z), and an image descriptor for each of the plurality of feature points may be included. Here, the image descriptor is for distinguishing a feature point on an image photographing a specific region from a feature point included in an image photographing another region, and a multidimensional vector in which the correlation between the feature point and the pixels surrounding the feature point is expressed as a vector. Can mean

The DB 300 may be a space for storing directly captured images or a space for storing map information provided from a portal site such as external Naver, Daum, and Google.

As shown in FIG. 2, the pose estimation apparatus 100 according to the embodiment may include a camera image collection unit 110.

The camera image collection unit 110 may collect a plurality of images. A plurality of images may be collected from the camera of the terminal.

The pose estimation apparatus 100 according to the embodiment may include a DB image collection unit 120.

The DB image collection unit 120 may collect a plurality of second images from the DB. The second image may be collected using GPS information of the terminal. The second image may be an image in which the location of the terminal is measured from GPS information, and the measured location of the terminal and the photographed location are closest. The second image is a 2D image, a position (u, v) of each of a plurality of features or feature points included in the 2D image on a 2D image, and an absolute 3D coordinate on the earth of each of the plurality of feature points ( x, y, z), and an image descriptor for each of the plurality of feature points may be included.

The pose estimating apparatus 100 according to the embodiment may include a first pose estimating unit 130.

The first pose estimating unit 130 may estimate a first pose of the camera by comparing the first image and a plurality of second images. The first pose estimating unit 130 may estimate a first pose of the camera by comparing the feature points of the first image and the feature points of the plurality of second images.

The pose estimating apparatus 130 according to the embodiment may include a second pose estimating unit 140.

The second pose estimating unit 140 may estimate a second pose of the camera using a plurality of first images. The second pose estimating unit 140 may estimate a second pose of the camera by comparing feature points of a plurality of first images with each other.

The pose estimating apparatus 100 according to the embodiment may include a third pose estimating unit 150.

The third pose estimating unit 150 may estimate a third pose of the camera using the first pose and the second pose. The first pose may take a large amount of computation as the DB image is compared with the first image using the earth coordinate system. Accordingly, the first pose may be measured every first period. Here, the first period may be 30Hz or 60Hz.

On the other hand, the second pose may have a small amount of computation because an area overlapping with the previous image exists as the plurality of first images are compared. Accordingly, the second pose may be measured every second period. Here, the second period may be several hundred ms to 1 sec.

The third pose estimating unit 150 may estimate a third pose by comparing images at the same viewpoint as the first pose and the second pose have different periods. On the other hand, when the first pose is not measured, the second pose may be estimated as the third pose.

The second image has an absolute position, but does not have a continuous characteristic. In addition, although the first image has a continuous characteristic, a camera draft may occur because the photographed position may change.

Accordingly, the third pose estimating unit 150 has an effect of being able to more accurately estimate the pose of the camera by supplementing each other's shortcomings by using the first pose and the second pose at the same viewpoint.

The third pose estimating unit 150 may use a Mahalanobis distance for a distance between the first pose and the second pose.

If the Mahalanobis distance is less than the first threshold value, the third pose may be estimated using an average value of the distance between the first pose and the second pose.

When the Mahalanobis distance is greater than a second threshold value greater than the first threshold value, the second pose may be used to estimate the third pose.

If the Mahalanobis distance is a value between the first threshold and the second threshold, the third pose may be estimated by reducing the weight of the second pose.

The pose estimation apparatus 100 according to the embodiment may further include an object augmentation unit 160.

The object augmentation unit 160 may determine a position to implement the virtual object on the screen displayed on the terminal using the third pose of the camera, for example, the position and direction of the camera. Here, the object augmentation unit may be omitted from the pose estimation apparatus.

Hereinafter, a pose estimation method according to an embodiment will be described with reference to FIGS. 3 to 5.

3 is a block diagram showing a pose estimation method according to an exemplary embodiment, FIG. 4 is a diagram for explaining a process of collecting a DB image according to an exemplary embodiment, and FIG. 5 is a step of estimating a third pose according to an exemplary embodiment. It is a figure for explaining.

Referring to FIG. 3, the pose estimation method according to the embodiment may perform an operation S100 of collecting a camera image.

In the step S100 of collecting camera images, a plurality of first images may be collected from the terminal. The first image may be an image obtained from a camera or sensor of the terminal.

In the step S100 of collecting the camera image, a plurality of first images captured while the terminal is moving may be collected. The first image may be a 2D image. The first image may be a still image at a specific point in time.

The pose estimation method according to the embodiment may perform the step (S200) of collecting a DB image.

In the step of collecting DB images (S200), a plurality of second images may be collected from the DB. As shown in FIG. 4, the second image 20 may be images close to the location of the first image 10 captured by the terminal using GPS information of the terminal. Accordingly, the second images 20 may be arranged and received in an order close to the terminal. For example, when the first image 10 is stored, the 2-1 image 21 closest to the terminal is received first, and the 2-2 image 22 closest to the second is received second, The fifth image 25, which is the fifth closest, may be finally received.

The second image 20 is a 2D image, a position (u, v) of each of a plurality of features or feature points included in the 2D image on the 2D image, and each of the plurality of feature points is an absolute It may include 3D coordinates (x, y, z) and an image descriptor for each of the plurality of feature points.

The pose estimation method according to the embodiment may perform an operation S300 of estimating a first pose.

In estimating the first pose (S300 ), the first pose of the camera may be estimated by comparing the first image and the plurality of second images. The first pose may be estimated by matching feature points of the first image and feature points of a plurality of second images. Here, since unnecessary feature points may be included in the first image and the second image, a step of removing unnecessary feature points may be further performed.

The pose estimation method according to the embodiment may perform an operation S400 of estimating a second pose.

In estimating the second pose (S400 ), a second pose of the camera may be estimated using a plurality of first images. In estimating the second pose (S400 ), the second pose of the camera may be estimated by matching feature points of a plurality of first images with each other.

The pose estimation method according to the embodiment may perform an operation S500 of estimating a third pose.

In estimating the third pose (S500 ), a third pose may be estimated using the first pose and the second pose.

As shown in FIG. 5, the first pose T1 may be measured every first period (30Hz, 60Hz), and the second pose T2 may be measured every second period (hundreds of ms to 1sec).

Accordingly, in estimating the third pose (S500 ), the third pose may be estimated using the first pose T1 and the second pose T2 at the same viewpoint. In the drawing, the third pose may be estimated at the first time point t6 and the second time point t11, but this is not limited.

Here, at the time points t2 to t5 and t7 to t10 at which the first pose T1 is not measured, the second pose T2 may be estimated as the third pose.

The third pose can be estimated using the Mahalanobis distance (Md). Here, the Mahalanobis distance Md may be defined by Equation 1.

[Equation 1]

Here, x denotes a position in the first pose, and y denotes a position in the second pose. S ^-1 means variance. Variance can be obtained by projecting 3D coordinate values on the image.

The third pose may be calculated by Equation 2. The third pose may vary depending on the condition of the Mahalanobis distance (Md). For example, the Mahalanobis distance (Md) is less than the first threshold (d1), the Mahalanobis distance (Md) is between the first threshold (d1) and the second threshold (d2), or Depending on the case where the distance Md is greater than the second threshold d2, the third pose may be calculated differently.

[Equation 2]

The third pose may be calculated by Equation 3.

[Equation 3]

Here, σ means the standard deviation, and μ means the average value. The square of the standard deviation means the variance value. Here, the average value may be obtained by using a 3D coordinate value in the image.

For example, when the average value of the first pose T1 is 2 and the weight is 4, the average value of the second pose T2 is 1 and the weight is 2, the third pose may be 5/3. Here, the weight may be an inverse value of variance.

That is, when the Mahalanobis distance Md is smaller than the first threshold d1, the first pose T1 and the second pose T2 may be estimated using an average value according to a weight.

In addition, when the Mahalanobis distance Md is between the first threshold value d1 and the second threshold value d2, T'local in Equation 2 may be calculated by Equation 4.

[Equation 4]

Here, v may be an arbitrary value for lowering the weight, and v may be greater than 1.

In addition, if the Mahalanobis distance Md is greater than the second threshold value d2, the second pose T2 value may be used.

In the embodiment, by using the first pose and the second pose, it is possible to estimate a more accurate camera pose by removing all cases in which the first pose is incorrect or the second pose is incorrect.

The pose estimation method according to the embodiment may perform the step (S600) of determining an implementation position of a virtual object.

In the determining (S600) of the implementation position of the virtual object, a position to implement the virtual object on the screen displayed on the terminal may be determined using the position and direction of the camera based on the third pose.

Although described above with reference to the drawings and embodiments, those skilled in the art will understand that the embodiments can be variously modified and changed without departing from the technical spirit of the embodiments described in the following claims. I will be able to.

100: pose estimation device
200: terminal
300: DB

Claims (10)

In a method performed in a pose estimation apparatus for estimating a pose of a camera provided in a terminal,
Collecting a plurality of first images from the camera;
Collecting a plurality of second images from a DB using GPS information of the terminal;
Estimating a first pose of the camera by comparing the first image of any one of the plurality of first images with the plurality of second images;
Estimating a second pose of the camera using the plurality of first images; And
Estimating a third pose of the camera by comparing the first pose and the second pose estimated at the same time point
Pose estimation method comprising a. The method of claim 1,
In the step of estimating the third pose of the camera,
The third pose is a pose estimation method using a Mahalanobis distance with respect to a distance between the first pose and the second pose. The method of claim 2,
In the step of estimating the third pose of the camera,
When the Maharanobis distance is less than a first threshold value, the third pose is estimated by using an average value according to a weight of the first pose and the second pose. The method of claim 3,
In the step of estimating the third pose of the camera,
When the Maharanobis distance is greater than a second threshold value greater than the first threshold value, the pose estimation method of estimating the third pose from the second pose. The method of claim 4,
In the step of estimating the third pose of the camera,
If the Maharanobis distance is a value between the first threshold and the second threshold, a weight is increased to the second pose to estimate the third pose. The method of claim 1,
In the step of estimating the third pose of the camera,
The first pose is measured every first period, and the second pose is measured every second period faster than the first period,
The third pose is a pose estimation method for estimating the second pose as the third pose when the first pose is not measured. The method of claim 1,
And after estimating the third pose, determining a position to implement a virtual object on the screen of the terminal using the third pose. A camera image collection unit that collects a plurality of first images from the camera;
A DB image collection unit that collects a plurality of second images from a DB using GPS information of the terminal;
A first pose estimating unit for estimating a first pose of the camera by comparing the first image of any one of the plurality of first images with the plurality of second images;
A second pose estimating unit for estimating a second pose of the camera using the plurality of first images; And
A third pose estimating unit that estimates a third pose of the camera by comparing the first pose and the second pose estimated at the same point in time
Pose estimation device comprising a. As a computer-readable recording medium storing a computer program,
The computer program,
A computer-readable recording medium comprising instructions for causing a processor to perform the method according to any one of claims 1 to 7. As a computer program stored in a computer-readable recording medium,
The computer program,
A computer program comprising instructions for causing said processor to perform a method according to claim 1.

Images