KR101546654B1 - Method and apparatus for providing augmented reality service in wearable computing environment - Google Patents

Abstract

The method includes generating local reference coordinates based on a three-dimensional image including depth information obtained through a camera, generating a local reference coordinate system based on the local reference coordinate system, dimensional world coordinate system of the 3D object in the world coordinate system and the rotation coordinate and the depth information of the user interrupt generation type of the 3D object selected in the 3D coordinate space integrated in the world coordinate system, And controlling the selected 3D object based on the sensed signal from the intra-terminal motion sensor providing the augmented reality service when searching for an execution menu for using the augmented reality service, A plurality of selectable hierarchical menu interfaces are displayed in the three-dimensional coordinate space It characterized in that it comprises the step of providing a new operation.

Description

Technical Field [0001] The present invention relates to a method and apparatus for providing an augmented reality service in a wearable augmented reality environment,

In a wearable augmented reality environment in which a user wearing an HMD (Head Mounted Display) creates a diglog space using a smart phone-based three-dimensional mobile input device and adds virtual content, Service provision.

2. Description of the Related Art [0002] Wearable glasses (see Fig. 2 (a)), which can be viewed anytime and anywhere in recent digital information, are reduced in weight and utilized to provide digital information (e.g., local information, photographs, , Restaurants, wikipedia, etc.) on the spot are immediately attracting much attention. This can serve as a new media that can replace portable Internet terminals (eg, smart phones, notebooks) used as media for acquiring existing information. In other words, it is possible to instantly provide information of interest to users in the field, and it is possible to instantly perform life logging services such as existing Facebook / Twitter / YouTube, and to acquire various information easily. This will have various ripple effects in social / cultural / economic / industrial aspects.

(1) augmented reality authoring interface

The existing augmented reality authoring method can be generally classified into low-level authoring based on programming and high-level authoring based on non-programming [6]. Programming-based authoring methods (such as ARToolKit, osgART, and Goblin XNA) have the advantage of being able to optimize performance through programming, but for technical developers who have expertise in computer vision and 3D graphics, It is difficult to do.

The non-programming based authoring method is based on graphical user interface (GUI) -based visual programming, and can be used by relatively unprofessional (ComposAR, CATOMIR, Metaio, etc.). However, it is difficult to apply the existing GUI based 2D interface to 3D space manipulation discussed in this study. On the other hand, a tangible user interface (TUI) based authoring method enables a more abstract and easy authoring than a GUI method, and can be operated in three dimensions (iaTAR, ARtalet, etc.). However, TUI is mainly useful for manipulating 3D objects in close proximity. Therefore, it is inappropriate as an interface of Digilog space which is aimed at the three dimensional space which is aimed in this study.

(2) For augmented reality authoring coordination  Way

The simplest matching method is to match the mirror world based on GPS / compass sensor and is used in most mobile augmented reality applications (Layar, Wikitude AR, Junaio, Sekai camera). It is possible to operate in outdoor environment, and accuracy of matching is very low due to error of GPS / compass sensor information.

  For more precise matching, there is a way to use 2D objects. Perform camera tracking and match virtual objects (ARToolKit, ARToolKit Plus, 2D barcode markers) using square markers with clear color contrast. Alternatively, there is a method of matching a 2D object of a natural image to a target (BazAR, Qualcomm AR SDK). You can also apply augmented reality plug-ins to commercial software (Virtools, Max / Maya, Unity3D, RhinoAR, Autodesk 3DS Max, Flash) to match software 3D models to 2D objects. However, this method is not suitable for the matching of the three-dimensional space covered in this study because the object of matching is limited to the two-dimensional plane.

(3) Wearable  How to manipulate 3D object based on mobile input device in augmented reality environment

The simplest way to manipulate a 3D object using a mobile input device is to press the button device of the mobile device. However, such discrete user input is not possible to perform high dimensional simultaneous manipulation (three dimensional movement, three dimensional rotation) in 3D space and time delay. As an extension of this, it is also impossible to perform operations with a high degree of freedom in a two-degree-of-freedom input method using a joystick or touch-based input. Using the built-in sensors of the mobile input device (eg, tilt, accelerometer, gyro, compass sensor, etc.) allows manipulation of 3D degrees of freedom and allows for gyroscopes, game controllers, menu selection, text input, scroll gestures, Respectively. However, they also can not be manipulated freely (movement / rotation of 6 degrees of freedom) in 3D space. Generally, an external tracking device must recognize / track the position / rotation of the input device (3D game controller, uWand, VR Wand, etc.) in order to operate a 3D object with 6 degrees of freedom. However, these input devices require an external tracking device It is impossible to apply it to an outdoor environment and a usual space.

Therefore, in order to solve such a problem, the present invention proposes a method of authoring a diglit space easily matching and inserting / manipulating a virtual content in a wearable computing environment. In other words, in the general indoor and outdoor environment without an external tracking system, the proposed method accurately matches the mirror world to the real space using the 3D user interface based on the smartphone, and then, the 3D virtual object is retrieved, The goal is to be able to do.

Specifically, a user who wears a video-see-through head mounted display (HMD) with a depth-of-view (RGB) depth of view (video) can view an arbitrary space The feature data can be acquired in real time, and the local reference coordinates (Local Reference Coordinates) for matching the virtual space can be automatically generated using the depth image information. This allows the user to constantly acquire Digilog space information / content even on free movement. In a wearable augmented reality environment, a user can call and manipulate 3D virtual objects at various distances using a smart phone-based 3D mobile input device. In particular, this patent is intended to modify or personalize the control gain of the mobile input device in consideration of the radius of the wrist motion of the user, thereby reducing the fatigue of the wrist and improving the manipulation performance. Further, the mobile device can be intuitively navigated / selected to suit the wearable augmented reality.

According to an aspect of the present invention, there is provided a method of generating a local reference coordinate system, the method comprising: generating local reference coordinates based on a three-dimensional image including depth information obtained through a camera; A step of integrating a world coordinate system of a mirror world which is a virtual space and a rotation and depth information of a user interrupt occurrence type of a 3D object selected in a three-dimensional coordinate space integrated in the world coordinate system, The method comprising the steps of: controlling the selected 3D object based on a range of motion (RoM) based on a signal detected from a motion sensor in the terminal providing the augmented reality service when searching for an execution menu for using the augmented reality service; A plurality of hierarchical menu interfaces capable of selecting a corresponding menu for each level, And displaying the menu in a space to provide a menu operation.

According to another aspect of the present invention, there is provided an image processing apparatus including a coordinate generating unit for generating a local reference coordinate system and a world coordinate system of a mirror world based on a three-dimensional image including depth information obtained through a camera, , Integrates the local reference coordinate system and the world coordinate system generated from the coordinate generating unit, and integrates the rotation and depth information of the user interrupt occurrence type of the 3D object selected through the user interface unit in the integrated three-dimensional coordinate space, And a menu operation unit for displaying a plurality of hierarchical menu interfaces in a three-dimensional coordinate space under the control of the control unit to provide a menu operation, characterized by comprising: do.

The present invention allows the user to obtain additional information on the object of interest or the situation actually observed in the field in the Digilog space, so that the understanding can be enhanced and the work can be performed more efficiently. It also has the effect of enabling users to have an interesting experience that goes beyond time and space in the field.

In addition, it is possible to invite a scene of a movie to the scene and experience it, to invite the past mirror world to reality, to travel in the past, and to bring the mirror world of the remote space to the scene, .

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an overall flowchart of a method for providing an augmented reality service in a wearable augmented reality environment according to an embodiment of the present invention.
FIG. 2 to FIG. 5 and FIG. 7 are diagrams illustrating a screen for providing an augmented reality service in a wearable augmented reality environment according to an embodiment of the present invention;
6 is a detailed block diagram of an augmented reality service providing apparatus in a wearable augmented reality environment according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It will be appreciated that those skilled in the art will readily observe that certain changes in form and detail may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims. To those of ordinary skill in the art.

In a wearable augmented reality environment in which a user wearing an HMD (Head Mounted Display) creates a diglog space using a smart phone-based three-dimensional mobile input device and adds virtual content, More specifically, a author who wears an RGB-D video transmission type HMD acquires image feature data of an arbitrary physical space, which is not previously known, in real time, The 3D virtual objects are retrieved at various distances using a smartphone-based mobile input device, and the 3D virtual objects are retrieved at various distances using a smartphone-based mobile input device. And manipulating the non-programmable devices Authoring method can also easily use a regular user, and also to provide a technology that can create a digital log space in our daily living space is not installed an external tracking device.

In particular, the DigiLog Space based on the wearable augmented reality considered in the present invention is a space in which a physical reality space and a mirror world, which is a replicated virtual space thereof, are matched with each other, and augmented reality glasses (HMD : Head Mounted Display) can acquire various information of the mirror world corresponding to the space in three dimensions [1]. That is, in Digilog space, the user can get additional information about the subject or situation that he / she is actually observing in the field, so that understanding can be enhanced and work can be performed more efficiently. Users can also have an interesting experience in the field, beyond time and space. For example, you can experience a scene of a movie by bringing it to the scene (Figure 2. (b)), or you can travel past past the mirror world as a reality. Or you can take a remote experience with the mirror world of the remote space. Thus, Digilog space has various application possibilities.

Hereinafter, a method of providing an augmented reality service in a wearable augmented reality environment according to an embodiment of the present invention will be described in detail with reference to Figs. 1 to 5.

1 is an overall flowchart of a method for providing an augmented reality service in a wearable augmented reality environment according to an embodiment of the present invention.

Referring to FIG. 1, a 3D image including depth information is acquired through a camera 110 in step 110. In step 112, a local reference coordinate system (Local) based on a 3D image including depth information acquired through the camera, Reference Coordinates).

At this time, the process of generating the local reference coordinate system acquires information about the plane viewed by the user for setting the 3-DoF (Degree of Freedom) rotation of the coordinate system using the following equation (1).

[Equation 1]

&Quot; (2) "

(여기서, Π: 평면, a, b, c, d: 평면을 구성하는 파라미터,

,

,

: 3차원 공간상의 한 점의 좌표,

,

: 평면상의 임의의 서로 다른 두 점, N : 계산할 점들의 개수)

(Where, Π: plane, a, b, c, d: parameters constituting the plane,

,

,

: Coordinates of a point on a three-dimensional space,

,

: Two arbitrary points on the plane, N: number of points to be calculated)

The information about the plane viewed by the acquired user can be defined as an optimization problem for obtaining the variables a, b, c, and d constituting the plane equation for the three-dimensional image feature coordinates as shown in equation (1) have.

From the solution of the equation (1), the normal vector perpendicular to the plane is calculated through the above equation (2) and set as the z-axis of the coordinate axis, and two arbitrary two points on the plane are selected, .

The rotation axis of the reference coordinate system for the actual space can be set as shown in FIG. 7 by calculating the y-axis of the coordinate axis through the set z-axis and the x-axis outer product.

Then, in order to determine the position of the reference coordinate system and correct the rotation axis set in an arbitrary direction, the following equation (4) is derived using the following equation (3) using a tangent line where the vertical plane and the horizontal plane are in contact with each other.

은 한 점

와, 방향 벡터

로 표현될 수 있으며, 두 평면 모두에 대해서 수직임을 알 수 있다. 이러한 관계를 이용하여 수학식 6을 유도할 수 있고,

는 두 평면에 모두 접하는 접점이기 때문에 하기의 수학식 5를 통해 두 평면 방정식의 합의 차이를 최소로 하는 점인

을 구하여 접점으로 설정한다. As shown in the following Equation 3,

A point

And a direction vector

, And it can be seen that it is perpendicular to both planes. Using this relationship we can derive Equation 6,

Is a contact which is tangential to both planes, it is a point that minimizes the difference of the sum of the two plane equations through the following equation

And sets it as a contact point.

로 설정하고, 상기 x축과 z축을 외적하여 레퍼런스 좌표축의 회전축을 도 7의 (b)와 같은 형태로 보정할 수 있다.And sets the x-axis, which is a horizontal axis of the set coordinate system, as a horizontal vector

, And the rotation axis of the reference coordinate axis can be corrected in the same manner as shown in FIG. 7 (b) by externally crossing the x-axis and the z-axis.

&Quot; (3) "

&Quot; (4) "

&Quot; (5) "

(여기서,

,

: 서로 다른 두 평면,

: 사용자가 지정한 3차원 점, t: 직선의 파라미터,

,

: 서로 다른 두 평면의 법선벡터,

: 임의의 3차원 점)

(here,

,

: Two different planes,

: A user-specified three-dimensional point, t: a parameter of a straight line,

,

: Normal vector of two different planes,

: Arbitrary three-dimensional point)

Finally, to unify the scale of the real space with the arbitrary scale of the coordinate system calculated from the RGB image, the distance between two different three-dimensional points restored is calculated as shown in Equation 6 below. Then, through the depth information, the distance is calculated for each of the two points as shown in Equation 7. Finally, we calculate the ratio of these two distances as shown in Eq. (8). This allows the virtual object to be augmented to the correct size by reflecting the unit scale of the real space (eg units of mm, cm, m, etc.).

&Quot; (6) "

&Quot; (7) "

&Quot; (8) "

(여기서,

,

: 임의 스케일인 가상 공간에서, 임의의 두 점,

,

: 카메라 중심 위치를 기준으로 한, 점

,

에 대한 실측 거리, dimage:임의 스케일인 가상 공간에서, 두 점 사이의 거리)

(here,

,

: In an arbitrary-scale virtual space, any two points,

,

: Point based on camera center position

,

The distance between two points in a virtual space of arbitrary scale)

Next, in step 114, the local reference coordinate system generated and the world coordinate system of the mirror world, which is a virtual space in which the actual space is copied, are integrated.

That is, the user acquires the image feature information of the geometric structure with respect to the site space that has not been previously modeled using the camera, and generates a local reference coordinate system based on the image feature information.

As shown in FIG. 3 (b), in order to solve the difficulty of smoothly performing three-dimensional manipulation frequently used in Digilog space, the existing two-dimensional user interface The coordinate system is matched as shown in Fig. 3 (a).

This is because the scales between the reality and the mirror world must be precisely unified so as to enhance the three-dimensional information / contents to an accurate size. Through the matching, the posture of the user's HMD camera and the posture of the camera of the mirror world coincide with each other, The point of view is the same as the point of view in the mirror world.

In step 116, a 3D object located in the three-dimensional coordinate space integrated in the world coordinate system is displayed.

의 크기가 t보다 작거나 존재하지 않는 경우, 상기 단말에 구비된 자이로 센서 정보

가 함수

에 의하여 변환된 3D 객체의 회전 정보

에,

의 크기가 t보다 클 때 수행되는 3D 객체의 이동 모션 벡터를 곱한 값을 이용하여 움직임이 인식된다.Here, the 3D object may include a motion vector of the user input means detected by the input unit of the terminal through Equation (9)

When the size of the gyro sensor information is smaller than t or does not exist,

Function

The rotation information of the 3D object converted by the

on,

The motion is recognized by multiplying the motion vector of the 3D object performed when the size of the 3D object is larger than t.

&Quot; (9) "

(여기서,

: 사용자 조작의 회전 정보,

: 객체 조작의 회전 정보,

: 객체 조작의 이동 정보)

(here,

: Rotation information of user operation,

: Rotation information of object manipulation,

: Movement information of object manipulation)

If the selection of the 3D object is made, the controller 120 proceeds to step 120 and recognizes the rotation and depth information based on the motion range of each object previously set. In step 122, the 3D object is controlled.

Here, the predetermined motion range for each object is obtained by obtaining the angle range to be operated within the maximum operation range for each object based on the information about the rotation range for each object of the user collected in advance, Lt; RTI ID = 0.0 > (18) < / RTI >

In addition, the user interrupt refers to a movement such as a contact with a user input means such as a finger, a stylus pen, or the like on the display on which Digilog space is displayed, or a rotation applied to the entire device. User interrupts are generated by differentiating according to the type of interrupt occurrence, for example, an inconvenient user moving in a specific motion range, a right or left-handed user, and an age (child, elderly) To improve performance and usability.

More specifically, regarding the rotation operation of the mobile device, if the rotation information of the device and the rotation information of the 3D object are mapped to 1: 1, if the user manipulates the robot beyond the maximum RoM, physical difficulties arise, Performance may be degraded.

Therefore, non-1: 1 mapping may show better performance, but it may be intuitively difficult to understand because the rotation operation of the 3D object is not the same as the rotation of the device. Therefore, it is important to determine the optimal RoM mapping that minimizes the cognitive burden while showing high performance.

Referring now to FIG. 4, FIG. 4 shows the RoM of the wrist. For tilt rotation, the RoM of the ulnar and radial deviation is known to be between 15 and 30. Roll rotation related to adduction and abduction of the wrist is between 65 and 60 degrees. Yaw rotation related to wrist flexion is known to be between 60 and 45 degrees.

At this time, the simplest rotation mapping function is expressed by Equation (10).

&Quot; (10) "

&Quot; (11) "

(여기서,

: 증강현실 디스플레이공간에서 가상객체의 회전 각도,

: 회전 각도 맵핑 함수,

: 사용자가 조작하는 회전 각도, a: 함수의 특성 변화를 위해 임의로 설정될 수 있는 상수,

: 손목의 운동 반경 강도,

: 장치의 초기 회전 각도,

,

: 사용자가 장치를 손에 쥐고 회전할 때의 최소 및 최대 각도 범위,

: 손목의 운동 반경 각도를 고려한 회전 매핑 결과)

(here,

: Rotation angle of a virtual object in augmented reality display space,

: Rotation angle mapping function,

: A rotation angle operated by the user, a: a constant that can be arbitrarily set for changing the characteristic of the function,

: The strength of the exercise radius of the wrist,

: Initial rotation angle of the device,

,

: The minimum and maximum angular range when the user holds the device in rotation,

: Rotation mapping result considering the angle of motion radius of the wrist)

는 사용자 조작공간 (Control Space)에서 모바일 장치의 회전각도이며

는 증강현실 디스플레이공간 (Display Space)에서 가상객체의 회전각도이다. 회전각도

는 회전 축 Tilt, Roll, Yaw 각각에 따라 독립적으로 적용된다. 그리고 a는 함수의 특성 변화를 위해 임의로 설정될 수 있는 상수이다. 이와 비슷하게 RoM을 고려할 경우, 수식 11처럼 매핑 함수를 정의할 수 있다. In Equation (10)

Is the rotation angle of the mobile device in the user control space (Control Space)

Is the rotation angle of the virtual object in the augmented reality display space. Rotation angle

Is independently applied to each of the rotation axes Tilt, Roll, and Yaw. And a is a constant that can be arbitrarily set to change the characteristics of the function. Similarly, when RoM is considered, a mapping function can be defined as in Equation 11.

은 문헌으로 보고된 손목의 운동 반경 각도이다. 그리고 회전축의 초기 각도와 개별 사용자의 개인화된 RoM을 고려할 경우, 하기의 수학식 12와 같이 매핑함수를 정의할 수 있다. 이를 위해서, 사전에 수집한 사용자의 손목 회전범위 정보를 기반으로 하여 손목의 최대 조작범위(정방향

/역방향

) 내에서, 조작하고자 하는 각도범위 알아내고 이를 수식에 적용한다. 즉. 사용자의 최대 손목 조작 각도 대비 사용자의 현재 입력 각도 비율을 이용하여, 사용자가 손목에 무리가 가지 않는 최대 손목 회전 조작 범위 내에서 가상객체의 360도 회전 각도를 쉽게 조작할 수 있도록 한다. here

Is the radius of motion of the wrist reported in the literature. When considering the initial angle of the rotation axis and the personalized RoM of the individual user, a mapping function can be defined as shown in Equation (12) below. For this purpose, based on the previously collected wrist rotation range information, the maximum operation range of the wrist (forward direction

/ Reverse direction

), Find the angle range you want to manipulate and apply it to the formula. In other words. The user can easily manipulate the 360 degree rotation angle of the virtual object within the maximum wrist rotation operation range in which the user does not feel uncomfortable on the wrist by using the ratio of the user's current input angle with respect to the maximum wrist operation angle of the user.

&Quot; (12) "

(여기서,

: 사용자 개개인의 손목 운동 반경을 고려한 회전 매핑 결과)

(here,

: Rotation mapping result considering the radius of wrist motion of each user)

Next, in step 124, it is checked whether or not the execution menu corresponding to the augmented reality service use is checked. In step 126, a plurality of hierarchical menu interfaces are displayed.

In operation 126, when a plurality of hierarchical menu interfaces are displayed in searching for an execution menu for using an augmented reality service, in operation 128, A plurality of hierarchical menu interfaces capable of menu selection are displayed in the three-dimensional coordinate space to provide menu operation.

The menu operation may include searching for a menu in the form of a pie using the rotation information of the terminal input device, selecting a specific partial area of the ARWand screen, The direction of the two-dimensional rotation is obtained based on the horizontal of the ARWand, and the selected menu is recognized through the following Equation (14).

This is because the conventional menu selection interface mainly searches for items through GUI-based tree-type menus and requires precise pointing in order to select a menu. However, since it is not suitable for use in a wearable augmented reality environment in which a user frequently moves, According to an embodiment of the present invention, a menu can be searched and selected using only the internal sensor information of the mobile input device so that the user can freely write while moving on the mobile input device based menu for the wearable augmented reality environment .

는 메뉴 레벨 i에서의 메뉴 아이템의 개수이며 항상 0보다 크다. 그리고 최소한 한 개의 메뉴 항목이 항상 선택 되도록 한다.More specifically, as shown in FIG. 5, the rotation information of the mobile input device is used to search for a menu of a Pie style, and selection is made to be activated by pressing a specific partial area (button) of the ARWand screen. To do this, the 2D rotation direction (roll) is determined based on the pitch of the ARWand from the motion sensor value as shown in Equation 13, and the number of the menu in the menu step is selected through Equation (14). Where H is the step function, where deg is a value from 0 to 360 degrees. And

Is the number of menu items at menu level i and is always greater than zero. And at least one menu item is always selected.

&Quot; (13) "

&Quot; (14) "

(여기서, H: 계단 함수(step function), Ni: 메뉴 레벨 i에서의 메뉴 아이템 개수)

(Where H: step function, Ni: number of menu items at menu level i)

In the foregoing, a method for providing an augmented reality service in a wearable augmented reality environment according to an embodiment of the present invention has been described.

Hereinafter, an augmented reality service providing apparatus in a wearable augmented reality environment according to an embodiment of the present invention will be described with reference to Fig.

6 is a detailed block diagram of an augmented reality service providing apparatus in a wearable augmented reality environment according to an embodiment of the present invention.

6, in the wearable augmented reality environment in which the present invention is applied, the augmented reality service providing apparatus includes an imaging unit 610, a user interface unit 612, a control unit 614, a coordinate generating unit 616, 618).

The coordinate generating unit 616 generates a local reference coordinate system and a mirror world coordinate system based on the three-dimensional image including the depth information obtained through the photographing unit 610 .

The coordinate generating unit 616 acquires information on a plane viewed by the user for setting a 3-DoF (Degree of Freedom) rotation of the coordinate system using Equation (15) below, Calculating a normal vector perpendicular to a plane through the following equation (15) and setting it as a z-axis of a coordinate axis, selecting two arbitrary two different points on the plane as an x-axis of a coordinate axis, and the y-axis of the coordinate axis is calculated through the external of the x-axis to generate the local reference coordinate system.

&Quot; (15) "

&Quot; (16) "

을 구하여 접점으로 설정하고, 상기 설정된 좌표계의 수평축인 x축을 두 평면이 접하는 수평 벡터 로 설정하고, 상기 x축과 z축을 외적하여 레퍼런스 좌표축의 회전축을 보정한다.In order to determine the position of the reference coordinate system and correct the rotation axis set in an arbitrary direction, the coordinate generating unit 616 uses the tangent line in which the vertical plane and the horizontal plane are in contact with each other, 18, and the difference between the sum of the two plane equations is minimized through the following equation

Is set as a contact point, and the x-axis, which is the horizontal axis of the set coordinate system, is set as a horizontal vector tangential to the two planes, and the rotation axis of the reference coordinate axis is corrected externally of the x-axis and the z-axis.

&Quot; (17) "

&Quot; (18) "

&Quot; (19) "

In order to unify the scale of the arbitrary scale of the coordinate system calculated from the RGB image with the scale of the actual space, the coordinate generating unit 616 may calculate the coordinates of the coordinates of two different three-dimensional points restored using the following equation (20) The distance is calculated for each of the two points through the depth information of the 3D image, and the ratio of the two distances is calculated as shown in equation (22).

&Quot; (20) "

&Quot; (21) "

&Quot; (22) "

The controller 614 controls the overall operation of the augmented reality service providing apparatus in the wearable augmented reality environment.

In addition, it is possible to integrate the local reference coordinate system and the world coordinate system generated from the coordinate generating unit 616, and calculate the rotation and depth information of the 3D object selected by the user interface unit 612 in the integrated three- Based on a predetermined range of motion (RoM), and controls the 3D object.

The control unit 614 acquires an angle range to be operated within the maximum operation range of each object based on the user-specific rotation range information collected by the user in advance, and obtains an optimal RoM mapping of the rotation information of the terminal and the rotation information of the 3D object The motion range for each object is recognized by defining a mapping function through the following equation.

The menu operation unit 618 displays a plurality of hierarchical menu interfaces in a three-dimensional coordinate space under the control of the control unit to provide a menu operation.

In addition, the menu operation unit 618 provides a menu in the form of a pie using the rotation information of the terminal input device, and when a specific partial area of the ARWand screen is selected through the user interface unit, Dimensional direction based on the horizontal of the ARWand from the motion sensor value of the formula, and performs a menu operation by recognizing the menu selected at the corresponding menu level by the following equation.

Although the exemplary embodiments of the present invention have been described for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. Without departing from the scope of the present invention. Accordingly, the scope of the present invention should not be limited by the illustrated embodiments, but should be determined by equivalents of the claims and the claims.

610: photographing section 612: user interface section
614: Controller 616: Coordinate generator
618:
[references]

Claims (15)

Generating local reference coordinates based on a three-dimensional image including depth information obtained through a camera;
Integrating the generated local reference coordinate system and a world coordinate system of a mirror world, which is a virtual space in which an actual space is copied,
A step of controlling the selected 3D object by recognizing the rotation and depth information for each user interrupt occurrence type of the 3D object selected in the three-dimensional coordinate space integrated in the world coordinate system on the basis of a predetermined range of motion (RoM) and,
A plurality of hierarchical menu interfaces capable of selecting a menu corresponding to a predetermined level based on a signal sensed by an intra-terminal motion sensor providing the augmented reality service when searching for an execution menu for using the augmented reality service, And providing a menu operation,
The process of providing the menu operation may include:
A search of a menu in the form of a pie using the rotation information of the terminal input device, a selection of a specific partial area of the ARWand screen, and a menu selection are performed. From the motion sensor value in the form of equation (31) And acquiring a two-dimensional rotation direction on the basis of the selected menu and recognizing a menu selected in the menu step through the following Equation (32).
&Quot; (31) "

(32)

(Where H: step function, Ni: number of menu items at menu level i) The method of claim 1, wherein the three-
Wherein the attitude information of the HMD (Head Mounted Display) camera is obtained by calculating the attitude information of the 3D geometrical structure with respect to the actual space not previously modeled. The method according to claim 1,
Wherein the generating the local reference coordinate system comprises:
A first step of obtaining information on a plane viewed by a user for setting a 3-DoF (Degree of Freedom) rotation of a coordinate system using Equation (23) below,
A second step of calculating the normal vector perpendicular to the plane from the solution of the equation (23) through the following equation (24) and setting the vector as the z axis of the coordinate axis,
A third step of selecting any two different points on the plane and setting the selected two points as the x axis of the coordinate axis;
And a fourth step of calculating a y-axis of a coordinate axis through an outer product of the z-axis and the x-axis set in the second step.
&Quot; (23) "

&Quot; (24) "

(Where, Π: plane, a, b, c, d: parameters constituting the plane,

,

,

: Coordinates of a point on a three-dimensional space,

,

: Two arbitrary points on the plane, N: number of points to be calculated) The method of claim 3,
After the fourth step, in order to determine the position of the reference coordinate system and correct the rotation axis set in an arbitrary direction, the following equation (26) is derived using the tangent line in which the vertical plane and the horizontal plane are in contact with each other using the following equation Step 5,
A point that minimizes a difference between the sum of two plane equations through the following equation (27)

And setting the contact point as a contact point;
The x-axis, which is the horizontal axis of the coordinate system set in the third step,

A step (e)
Further comprising an eighth step of correcting a rotation axis of a reference coordinate axis externally of the x-axis and the z-axis in the wearable augmented reality environment.
&Quot; (25) "

&Quot; (26) "

&Quot; (27) "

(here,

,

: Two different planes,

: A user-specified three-dimensional point, t: a parameter of a straight line,

,

: Normal vector of two different planes,

: Arbitrary three-dimensional point) 5. The method of claim 4,
After the eighth step, in order to unify the scale of the arbitrary scale of the coordinate system calculated from the RGB image and the scale of the actual space, a method of calculating distances between two different three-dimensional points restored using the following equation Step 9,
A tenth step of calculating a distance for each of two points through the depth information of the three-dimensional image as shown in the following equation (29)
And calculating a ratio of two distances according to Equation (30). ≪ EMI ID = 30.0 >
&Quot; (28) "

&Quot; (29) "

&Quot; (30) "

(here,

,

: In an arbitrary-scale virtual space, any two points,

,

: Point based on camera center position

,

The distance between two points in a virtual space of arbitrary scale) The method according to claim 1,
Wherein the 3D object comprises:
The motion vector of the user input means detected by the input unit of the terminal

When the size of the gyro sensor information is smaller than t or does not exist,

Function

The rotation information of the 3D object converted by the

on,

Wherein the motion is recognized using a value obtained by multiplying a moving motion vector of a 3D object performed when the size of the 3D object is larger than t.

(here,

: Rotation information of user operation,

: Rotation information of object manipulation,

: Movement information of object manipulation) 2. The method according to claim 1,
In order to obtain the angle range to be operated within the maximum operation range of each object based on the information on the rotation range of each object collected by the user in advance and to determine the optimum RoM mapping of the rotation information of the terminal and the rotation information of the 3D object, Wherein the mapping function is defined by a mathematical expression.

(here,

: Rotation angle of a virtual object in augmented reality display space,

: Rotation angle mapping function,

: A rotation angle operated by the user, a: a constant that can be arbitrarily set for changing the characteristic of the function,

: The strength of the exercise radius of the wrist,

: Initial rotation angle of the device,

,

: The minimum and maximum angular range when the user holds the device in rotation,

: Rotational mapping considering the angle of motion of the wrist,

: Rotation mapping result considering the radius of wrist motion of each user) delete A coordinate generating unit for generating a local reference coordinate system and a world coordinate system of a mirror world based on a three-dimensional image including depth information obtained through a camera,
And integrates the local reference coordinate system and the world coordinate system generated from the coordinate generating unit and integrates the rotation and depth information for each user interrupt occurrence type of the 3D object selected through the user interface unit in the integrated three dimensional coordinate space into a motion range of Motion: RoM) and controlling the 3D object,
And a menu operation unit for displaying a plurality of hierarchical menu interfaces in a three-dimensional coordinate space under the control of the control unit to provide a menu operation,
The menu operation unit,
A pie-shaped menu is provided using the rotation information of the terminal input device, and when a specific partial area of the ARWand screen is selected through the user interface unit, the motion sensor value of the form (41) And a menu operation is performed by acquiring a two-dimensional rotation direction based on the pitch of the ARWand and recognizing a menu selected in the menu step through the following equation (42) Augmented reality service providing device.
(41)

(42)

(Where H: step function, Ni: number of menu items at menu level i) The method of claim 9, wherein the plurality of hierarchical menu interfaces comprise:
Wherein the interface is a menu interface capable of selecting a menu corresponding to a preset level based on a signal sensed by an intra-terminal motion sensor providing the augmented reality service when searching for an execution menu for using the augmented reality service. A real service providing device. 10. The apparatus according to claim 9,
Information about the plane viewed by the user for setting the 3-DoF (Degree of Freedom) rotation of the coordinate system is obtained using the following equation (33), and from the solution of equation (33) Calculating a vertical normal vector as a z-axis of the coordinate axis, selecting two arbitrary two different points on the plane to set as the x-axis of the coordinate axis, and determining the y-axis of the coordinate axis through the set z- And generates the local reference coordinate system by calculating the local reference coordinate system in the wearable augmented reality environment.
&Quot; (33) "

&Quot; (34) "

(Where, Π: plane, a, b, c, d: parameters constituting the plane,

,

,

: Coordinates of a point on a three-dimensional space,

,

: Two arbitrary points on the plane, N: number of points to be calculated) 12. The apparatus according to claim 11,
In order to determine the position of the reference coordinate system and correct the rotation axis set in an arbitrary direction, the following equation (36) is derived using the following equation (35) using the tangent line where the vertical plane and the horizontal plane are in contact with each other, To minimize the sum of the two plane equations

And sets the x-axis, which is the horizontal axis of the set coordinate system, as a horizontal vector

And corrects the rotation axis of the reference coordinate axis by externally crossing the x-axis and the z-axis in the wearable augmented reality environment.
&Quot; (35) "

&Quot; (36) "

&Quot; (37) "

(here,

,

: Two different planes,

: A user-specified three-dimensional point, t: a parameter of a straight line,

,

: Normal vector of two different planes,

: Arbitrary three-dimensional point) 12. The apparatus according to claim 11,
In order to unify the scale of the arbitrary scale of the coordinate system calculated from the RGB image and the scale of the actual space, the distance between two different three-dimensional points restored by using the following expression (38) The distance is calculated for each of the two points as shown in Equation 39 below, and the ratio of the two distances is calculated as shown in Equation (40).
&Quot; (38) "

[Equation 39]

[Equation 40]

(here,

,

: In an arbitrary-scale virtual space, any two points,

,

: Point based on camera center position

,

The distance between two points in a virtual space of arbitrary scale) 10. The apparatus according to claim 9,
In order to obtain the angle range to be operated within the maximum operation range of each object based on the information on the rotation range of each object collected by the user in advance and to determine the optimum RoM mapping of the rotation information of the terminal and the rotation information of the 3D object, Wherein the mapping function is defined by a mathematical expression to recognize the predetermined motion range for each object in the wearable augmented reality environment.

(here,

: Rotation angle of a virtual object in augmented reality display space,

: Rotation angle mapping function,

: A rotation angle operated by the user, a: a constant that can be arbitrarily set for changing the characteristic of the function,

: The strength of the exercise radius of the wrist,

: Initial rotation angle of the device,

,

: The minimum and maximum angular range when the user holds the device in rotation,

: Rotational mapping considering the angle of motion of the wrist,

: Rotation mapping result considering the radius of wrist motion of each user) delete

Images