KR101258839B1 - Apparatus and method for displaying images - Google Patents

Abstract

PURPOSE: An apparatus and a method for displaying images are provided to calculate motion information by using homography calculated on the basis of feature points, thereby calculating the movement of a user without an additional sensing device. CONSTITUTION: A motion information calculator(250) calculates motion information by using homography calculated on the basis of extracted feature points. An image output unit(290) outputs the received scene image. [Reference numerals] (210) Photographing unit; (230) Feature point extraction unit; (250) Motion information calculator; (270) Communication unit; (290) Image output unit

Description

Image display device and method {Apparatus and method for displaying images}

The present invention relates to an image display apparatus and method, and more particularly, to estimate camera motion information using homography, and output a scene image obtained based on camera motion information in a panoramic image. It relates to an apparatus and a method.

Research into a tele-presence system that allows a user wearing a head mounted display (HMD) to feel as though they are in a remote place other than the space where they are currently located is being actively conducted. Among the technologies applied to the telepresence system, the performance of the technology for estimating the user's movement has the most influence on the performance of the telepresence system.

However, the conventional motion estimation technique estimates the user's motion by using a separate sensor such as a magnetic sensor, an inertial sensor, etc., which are provided in most HMDs. Accordingly, since a separate sensor is required to estimate the user's motion information, more cost is required.

Therefore, it is necessary to develop a technology capable of estimating a user's movement using only the HMD without a separate sensor.

KR 10-1998-0023727 (Daewoo Electronics Co., Ltd.) July 6, 1998 Patent Literature 1 is a virtual reality system for displaying a panoramic image. Patent Literature 1 calculates the position and orientation of a user from a user's motion signal. Then, the content of displaying the panoramic image corresponding to the calculated position and direction is disclosed. KR 10-2011-0074519 (Inter Touch Technologies, Inc.) June 30, 2011 Patent Document 2 is a telepresence robot with a camera boom, and Patent Document 2 operates a mobile robot in response to a command received from a remote control station. The present invention discloses a method of transmitting and displaying images captured by an auxiliary camera connected to a boom of a mobile robot to a remote control station.

SUMMARY An aspect of the present invention is to provide an image display apparatus and method for estimating camera motion information using homography and outputting a scene image obtained based on camera motion information in a panoramic image. To provide.

According to an aspect of the present invention, there is provided an image display device including: a photographing unit configured to photograph an image in a three-dimensional space in which a marker is installed on a plane; A feature point extracting unit extracting feature points from markers included in the image photographed by the photographing unit; A motion information calculator configured to calculate motion information using a homography calculated based on the feature points extracted by the feature point extractor; The motion information calculated by the motion information calculation unit is transmitted to a remotely photographing device, and is obtained based on the motion information in a panoramic image photographed by the photographing device based on the motion information. A communication unit which receives a scene image from the photographing apparatus; And an image output unit configured to output the scene image received through the communication unit.

According to another aspect of the present invention, there is provided a method of displaying an image, the method including: photographing an image in a three-dimensional space in which a marker is installed on a plane; Extracting feature points from markers included in the captured image; Calculating motion information using homography calculated based on the extracted feature points; Transmitting the calculated movement information to a photographing apparatus located remotely; Receiving, from the photographing apparatus, a scene image obtained based on the motion information in the panoramic image photographed by the photographing apparatus based on the motion information; And outputting the received scene image.

A computer-readable medium according to the present invention for achieving the above technical problem records a program for causing a computer to execute any one of the above methods.

According to the image display device and method according to the present invention, by calculating the motion information using a homography (homography) calculated based on the feature points extracted from the marker included in the image, the user's movement in real time without additional sensing device Can be calculated In addition, since real-time estimation of motion information is possible, scene images according to the user's movement can be output in real time, thereby providing a high-performance telepresence system that can give a feeling of being in a real space. Can be.

1 is a block diagram illustrating a video display device according to an exemplary embodiment of the present invention;
2 is a block diagram showing in more detail the configuration of a video display device according to an embodiment of the present invention;
3 is a view showing an example of a marker installed on a plane of a three-dimensional space according to a preferred embodiment of the present invention;
4 is a diagram illustrating an example of an image photographed by an image display device according to an exemplary embodiment of the present invention;
5 is a view for explaining a process of calculating motion information according to an embodiment of the present invention;
6 is a view showing in more detail the configuration of the photographing apparatus used in the preferred embodiment of the present invention;
7 is a view for explaining a process of generating a scene image from a panoramic image according to an embodiment of the present invention, and
8 is a flowchart illustrating an image display method according to an exemplary embodiment of the present invention.

Hereinafter, exemplary embodiments of a video display device and method according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a video display device according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the image display apparatus 100 is connected to a photographing apparatus 200 located remotely through the communication network 300. Meanwhile, the photographing apparatus 200 may be mounted on a mobile device (not shown) such as a robot, a vehicle, or the like.

The video display device 100 transmits motion information to the photographing device 200 when the user moves or rotates the image. The image display apparatus 100 outputs a scene image received from the photographing apparatus 200. The video display device 100 is a display device worn by a user on the head. The image display apparatus 100 may include a head mounted display (HMD), a helmet mounted sights (HMS), and the like.

The photographing apparatus 200 includes an omnidirectional camera, and acquires a panoramic image by capturing a surrounding scene through the omnidirectional camera. In this case, the photographing apparatus 200 may acquire the panoramic image after moving based on the motion information received from the image display apparatus 200.

In addition, the photographing apparatus 200 obtains a scene image corresponding to the motion information from the panoramic image by using the motion information received from the image display apparatus 100. The photographing apparatus 200 transmits the obtained scene image to the image display apparatus 100.

The communication network 300 includes not only a data network including a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), the Internet, but also a broadcasting network and a telephone network. It can include, wire and wireless, any communication method may be used.

2 is a block diagram illustrating in detail the configuration of a video display device according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the image display apparatus 100 includes a photographing unit 210, a feature point extracting unit 230, a motion information calculating unit 250, a communication unit 270, and an image output unit 290.

The photographing unit 210 captures an image in a three-dimensional space in which a marker is installed on a plane. Here, the marker provided on the plane constituting the three-dimensional space represents a marker according to ARToolKit. ARToolKit is an open library provided for implementing marker-based augmented reality (AR). In addition, the photographing unit 210 provides the captured image to the feature point extractor 230.

3 is a diagram illustrating an example of a marker installed on a plane of a three-dimensional space according to a preferred embodiment of the present invention.

As shown in FIG. 3, a plurality of markers MK are provided on a plane constituting a three-dimensional space. The photographing unit 210 is located inside the three-dimensional space to photograph the user's surroundings according to the direction of the user's gaze wearing the image display device 100.

4 is a diagram illustrating an example of an image photographed by an image display device according to an exemplary embodiment of the present invention.

Referring to FIG. 4, the image photographed by the photographing unit 210 according to the eyeline direction of the user wearing the image display apparatus 100 includes a marker installed on a plane constituting a three-dimensional space. For example, as illustrated in FIG. 4A, the images photographed by the photographing unit 210 may include a plurality of markers installed on the same plane. In addition, as illustrated in FIGS. 4B and 4C, the image photographed by the photographing unit 210 may include a plurality of markers installed on different planes.

The feature point extractor 230 extracts a feature point from a marker included in the image captured by the photographing unit 210. In addition, the feature point extractor 230 provides the extracted feature point to the motion information calculator 250.

5 is a diagram for explaining a process of calculating motion information according to an exemplary embodiment of the present invention.

The motion information calculator 250 calculates a homography based on the feature points extracted by the feature point extractor 230.

에서의 마커(MK)의 특징점

과 카메라 좌표계

에서의 촬영부(210)가 위치하는 점

사이의 기하학적 관계는 아래의 [수학식 1]과 같다. 여기서, 특징점

의 좌표는

로 표현되고, 점

의 좌표는

로 표현된다.Referring to FIG. 5, the marker coordinate system

Feature of markers (MK)

And camera coordinate system

Where the photographing unit 210 is located at

The geometric relationship between them is shown in Equation 1 below. Where feature points

Coordinates are

Expressed as

Coordinates are

Lt; / RTI >

은 마커 좌표계와 카메라 좌표계의 기하 관계를 나타내며, 아래의 [수학식 2]와 같다.here,

Represents a geometric relationship between the marker coordinate system and the camera coordinate system, and is represented by Equation 2 below.

은 동차(homogeneous) 좌표계에서의 카메라의 회전을 나타낸다.

은 동차 좌표계에서의 카메라의 이동을 나타낸다.here,

Denotes the rotation of the camera in a homogeneous coordinate system.

Represents the movement of the camera in the homogeneous coordinate system.

은 2차원 영상의 스크린 좌표계

에서의 점

로 아래의 [수학식 3]에 의해 사영(projection)된다.Assuming that the Z axis of the marker MK provided on the plane constituting the three-dimensional space is 0,

Is the screen coordinate system for two-dimensional images

Point at

Is projected by Equation 3 below.

및

는 점

의 좌표

를 나타낸다.

는 카메라 내부(intrinsic) 파라미터를 나타내며, 아래의 [수학식 4]와 같다.

는 호모그래피를 나타내며, 아래의 [수학식 5]와 같다.here,

And

Point

Coordinates of

Indicates.

Represents an intrinsic parameter of the camera, and is represented by Equation 4 below.

Represents homography, and is represented by Equation 5 below.

및

는 초점 거리를 나타낸다.

는 기울어짐(skew)을 나타낸다.

및

는 2차원 영상의 중심점의 좌표

를 나타낸다.

는 정칙행렬로서 역행렬이 존재한다.here,

And

Indicates the focal length.

Denotes a skew.

And

Is the coordinate of the center point of the 2D image.

Indicates.

Is an inverse matrix as a regular matrix.

The motion information calculator 250 calculates motion information through the following Equation 6 using the calculated homography.

은 x축의 회전 변환값을 나타낸다.

는 y축의 회전 변환값을 나타낸다.

은 회전행렬의 직교 성질을 이용하여

과

의 외적(cross product)으로 계산된다.

는 이동 변환값을 나타낸다.

는 행렬의 크기 인자(scale factor)를 나타내며, 아래의 [수학식 7]에 의해 계산된다.here,

Represents the rotation conversion value of the x-axis.

Represents the rotation conversion value of the y-axis.

Uses the orthogonal nature of the rotation matrix

and

It is calculated as the cross product of.

Denotes a shift conversion value.

Represents a scale factor of the matrix and is calculated by Equation 7 below.

As such, the motion information calculator 250 may calculate motion information including rotation information and motion information by using homography calculated based on the extracted coordinate values of the feature points.

The motion information calculator 250 may apply an extended kalman filter to the calculated motion information.

Meanwhile, when the feature point is a feature point extracted from the plurality of markers, the motion information calculator 250 may calculate a homography for each of the plurality of markers based on the feature points extracted from each of the plurality of markers. The motion information calculator 250 may calculate partial motion information for each of the plurality of markers based on homography for each of the plurality of markers. The motion information calculator 250 may calculate motion information by averaging partial motion information of each of the plurality of markers. In this case, the motion information calculator 250 may calculate the motion information by weighting the partial motion information of each of the plurality of markers using the area of each of the plurality of markers calculated based on the feature points extracted from each of the plurality of markers. have. Here, the larger the area, the higher the weight.

Of course, when the feature point is a feature point extracted from the plurality of markers, the motion information calculator 250 may calculate the motion information using the homography calculated based on the feature points extracted from the plurality of markers.

In addition, when the feature point is a feature point extracted from a plurality of markers installed on different planes, the motion information calculator 250 extracts the homography for each plane from the markers installed on the same plane among the plurality of markers. Can be calculated based on The motion information calculator 250 may calculate partial motion information of each plane based on homography for each plane. The motion information calculator 250 may calculate motion information by averaging partial motion information of each plane. In this case, the motion information calculator 250 may calculate the motion information by weighting the partial motion information of each of the planes using the area of each of the plurality of markers calculated based on the feature points extracted from each of the plurality of markers.

Here, when a plurality of markers are included in one plane, the extracted feature points and three-dimensional information are composed of linear equations and a direct linear transform (DLT) method is applied. Here, the DLT method is a method of calculating a projection matrix of a camera through a correspondence between a feature point detected in an image and three-dimensional points existing in the real world, and at least six corresponding points are required. After constructing the correspondence as a single matrix and performing singular value decomposition (SVD), the projective matrix is estimated with a vector corresponding to the minimum eigenvalue of the eigen vector matrix.

와 이를 영상 내 투영한 좌표에 대한 카메라 투영 행렬

를 포함한 함수

로 구성된다. 그리고 아래의 [수학식 8]과 같이 정의된 오차함수를 통해 실제 점

와의 차이를 최소화한다. 본 발명에서는 실시간에 적용하기 위해 누적 조건을 검사하여 키 프레임을 해석하는 지역(local) 번들 조정을 사용한다.In addition, when three different planes are included in an image, a LM (Levenberg-Marquadt) algorithm is used to consider a nonlinear motion characteristic of a user. Bundle adjustment is a method of accumulating motion information over several frames.

And the camera projection matrix for the projected coordinates in the image

Function

It consists of. And the actual point through the error function defined as [Equation 8] below

Minimize the difference from In the present invention, a local bundle adjustment is used to check a cumulative condition and interpret a key frame for real time application.

The communicator 270 transmits the motion information calculated by the motion information calculator 250 to the photographing apparatus 200 located remotely. In addition, the communication unit 270 receives from the photographing apparatus 200 a scene image obtained based on the motion information in the panoramic image photographed by the photographing apparatus 200 based on the motion information.

The image output unit 290 outputs a scene image received from the photographing apparatus 200 through the communication unit 270.

6 is a view showing in more detail the configuration of the photographing apparatus used in the preferred embodiment of the present invention.

Referring to FIG. 6, the photographing apparatus 20 includes a motion information receiver 610, a movement controller 620, a photographer 650, a scene image generator 670, and a scene image transmitter 690.

The motion information receiver 610 receives motion information from the video display device 100.

The movement controller 620 controls the movement of the photographing apparatus 20 based on the movement information received by the movement information receiver 610. For example, when it is determined that the position is changed through the movement information included in the movement information, the movement controller 620 moves the photographing apparatus 20 based on the movement information.

The photographing unit 650 includes an omnidirectional camera and acquires a panoramic image by photographing a surrounding scene through the omnidirectional camera. Of course, the photographing unit 650 may include a plurality of cameras photographing some directions, and may acquire a panoramic image by combining images photographed by the plurality of cameras.

The scene image generator 670 generates a scene image from the panorama image photographed by the photographing unit 650 based on the motion information received by the motion information receiver 610. That is, the scene image generator 670 maps the panorama image photographed by the photographing unit 650 to an environment map of an arbitrary spherical surface. The scene image generator 670 obtains a scene image by selecting an image region corresponding to 45 degrees in a diagonal direction from the panoramic image based on the motion information received by the motion information receiver 610.

7 is a diagram for describing a process of generating a scene image from a panoramic image according to an exemplary embodiment of the present invention.

Referring to FIG. 7, the scene image generating unit 670 may perform a scene based on the motion information received by the motion information receiving unit 610 in the panoramic image captured by the photographing unit 650 (FIG. 7A). An image (FIG. 7B) is generated.

The scene image transmitter 690 transmits the scene image generated by the scene image generator 670 to the image display apparatus 100.

8 is a flowchart illustrating an image display method according to an exemplary embodiment of the present invention.

The image display apparatus 100 captures an image in a three-dimensional space in which a marker is installed on a plane (S810). The image display apparatus 100 extracts a feature point from a marker included in the captured image (S820), and calculates motion information by using homography calculated based on the extracted feature point (S830). Here, since the process of calculating motion information using homography has been described above, a detailed description thereof will be omitted. Thereafter, the image display apparatus 100 transmits the calculated motion information to the photographing apparatus 200 (S840). In this case, the image display apparatus 100 may apply the extended Kalman filter to the calculated motion information and transmit the motion information to which the extended Kalman filter is applied to the photographing apparatus 200.

The photographing apparatus 200 that receives the motion information from the image display apparatus 100 captures a panorama image based on the motion information (S850). In this case, the photographing apparatus 200 may photograph the panoramic image after moving based on the motion information. The photographing apparatus 200 obtains a scene image from the panoramic image based on the motion information in operation S860, and transmits the obtained scene image to the image display apparatus 100 in operation S870.

The image display device 100 receiving the scene image from the photographing apparatus 200 outputs the scene image (S880).

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer apparatus is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like in the form of a carrier wave . In addition, the computer-readable recording medium may be distributed to computer devices connected to a wired / wireless communication network, and a computer-readable code may be stored and executed in a distributed manner.

Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the specific preferred embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the following claims. Anyone of ordinary skill in the art of various modifications can be made, of course, such changes are within the scope of the claims.

100: video display device, 200: photographing device,
300: communication network, 210: the photographing unit,
230: feature point extractor, 250: motion information calculator,
270: communication unit, 290: video output unit

Claims (10)

A photographing unit which photographs an image in a three-dimensional space in which a marker is installed on a plane;
A feature point extracting unit extracting feature points from markers included in the image photographed by the photographing unit;
A motion information calculator configured to calculate motion information using a homography calculated based on the feature points extracted by the feature point extractor;
The motion information calculated by the motion information calculation unit is transmitted to a remotely photographing device, and is obtained based on the motion information in a panoramic image photographed by the photographing device based on the motion information. A communication unit which receives a scene image from the photographing apparatus; And
And an image output unit configured to output the scene image received through the communication unit. The method of claim 1,
When the feature point is a feature point extracted from a plurality of markers, the motion information calculation unit calculates the homography for each of the plurality of markers based on the feature points extracted from each of the plurality of markers, and calculates for each of the plurality of markers. And calculating partial motion information based on the homography for each of the plurality of markers and averaging the partial motion information for each of the plurality of markers. The method of claim 2,
The motion information calculation unit calculates areas of each of the plurality of markers based on feature points extracted from each of the plurality of markers, and uses the areas of each of the plurality of markers, and the partial motion information of each of the plurality of markers. And calculating the motion information by weighted average. The method of claim 1,
The motion information calculator, when the feature point is a feature point extracted from a plurality of markers, calculates the homography based on feature points extracted from the plurality of markers, and calculates the motion information based on the homography. Video display device. 5. The method of claim 4,
The motion information calculator is configured to extract the homography for each plane from the markers on the same plane among the plurality of markers when the feature points are feature points extracted from the plurality of markers on different planes. And calculate partial motion information of each of the planes based on the homography of each of the planes, and calculate the motion information by averaging the partial motion information of each of the planes. Display device. The method according to any one of claims 1 to 5,
And the motion information calculator applies an extended kalman filter to the calculated motion information. Photographing an image in a three-dimensional space in which a marker is installed on a plane;
Extracting feature points from markers included in the captured image;
Calculating motion information using homography calculated based on the extracted feature points;
Transmitting the calculated movement information to a photographing apparatus located remotely;
Receiving, from the photographing apparatus, a scene image obtained based on the motion information in the panoramic image photographed by the photographing apparatus based on the motion information; And
And outputting the received scene image. 8. The method of claim 7,
The motion information calculation step,
If the feature point is a feature point extracted from a plurality of markers, calculating the homography for each of the plurality of markers based on the feature points extracted from each of the plurality of markers;
Calculating partial motion information for each of the plurality of markers based on the homography for each of the plurality of markers; And
And calculating the motion information by averaging the partial motion information for each of the plurality of markers. 8. The method of claim 7,
The motion information calculation step,
Calculating the homography based on the feature points extracted from the plurality of markers when the feature points are the feature points extracted from the plurality of markers; And
And calculating the motion information based on the homography. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the video display method according to any one of claims 7 to 9.

Images