WO2018212437A1

WO2018212437A1 - Calibration method for matching of augmented reality objects and head mounted display for performing same

Info

Publication number: WO2018212437A1
Application number: PCT/KR2018/002921
Authority: WO
Inventors: 박재완; 권기석; 김승균
Original assignee: 주식회사 맥스트
Priority date: 2017-05-19
Filing date: 2018-03-13
Publication date: 2018-11-22
Also published as: KR101894846B1

Abstract

A calibration method and a head mounted display for performing the same are disclosed. The calibration method according to the disclosed embodiment is performed in an optical see-through type head mounted display device comprising one or more processors and a memory for storing one or more programs executed by the one or more processors, the method comprising the steps of: displaying a plurality of virtual markers on the screen of a head mounted display; confirming whether the virtual markers and a plurality of markers positioned in front of the head mounted display are matched on the screen; calculating the angle between the markers on the basis of a camera of the head mounted display, if the virtual markers and the markers are matched on the screen; calculating the distance between the virtual markers displayed on the screen; and calculating the viewing angle at which a user views the screen, on the basis of the angle between the markers on the basis of the camera, the distance between the virtual markers, and the size of the screen.

Description

Calibration method for registration of augmented reality objects and head mounted display for performing the same

Embodiments of the present invention relate to optical see-through type head mounted display technology.

In a video see-through environment, augmented reality systems use the video camera itself as a tracking sensor, eliminating the need for a separate calibration process. However, in an optical see through environment, the user directly looks at the real world through a screen attached to a head mounted display (HMD). At this time, the field of view of the user's eyes and the field of view of the camera of the head-mounted display are different, and the physical position between the user's eyes and the camera of the head-mounted display is different. In order to do this, a correction is required.

Embodiments of the present invention provide a calibration method for correcting a difference in a viewing angle between a user's eyes and a camera in an optical see-through environment, and a head mounted display for performing the same.

A calibration method according to an embodiment of the present disclosure is a method performed in an optical see-through type head mounted display device having one or more processors and a memory storing one or more programs executed by the one or more processors. Displaying a plurality of virtual markers on a screen of the head mounted display; Checking whether the virtual markers and the plurality of markers located in front of the head mounted display match on the screen; Calculating an angle between the markers based on a camera of the head mounted display when the virtual markers and the markers match on the screen; Calculating a distance between the virtual markers displayed on the screen; And calculating a viewing angle viewed by the user through the screen based on the angle between the markers relative to the camera, the distance between the virtual markers, and the size of the screen.

The plurality of markers may include a first marker provided on one surface of a base portion positioned in front of the head mounted display, and a second marker spaced apart from the first marker in a horizontal or vertical direction from one surface of the base portion. The virtual marker may include a first virtual marker corresponding to the first marker and a second virtual marker corresponding to the second marker.

The calculating of angles between the markers may include calculating a horizontal marker angle or a vertical marker angle between the first marker and the second marker, and calculating a distance between the virtual markers. A horizontal virtual marker distance or a vertical virtual marker distance between the second virtual markers may be calculated.

The calculating of the viewing angle viewed by the user through the screen may include calculating a horizontal viewing angle viewed by the user through the screen based on the horizontal marker angle, the horizontal virtual marker distance, and the width of the screen. It may include.

The horizontal viewing angle FoVx viewed by the user through the screen may be calculated through the following equation.

(Mathematical formula)

After calculating the horizontal viewing angle FoVx, the method further includes converting the viewing angle of the camera to the viewing angle viewed by the user through the screen using the conversion matrix M according to the following equation. Can be.

(Mathematical formula)

ratio: screen height / screen width

far: Forward near-field constant of the camera relative to the camera

near: Forward distance constant of the camera relative to the camera

The calculating of the viewing angle viewed by the user through the screen may include calculating a vertical viewing angle viewed by the user through the screen based on the vertical marker angle, the vertical virtual marker distance, and the height of the screen. It may include.

The vertical viewing angle FoVy viewed by the user through the screen may be calculated by the following equation.

(Mathematical formula)

After calculating the vertical viewing angle FoVy, the method further comprises converting the viewing angle of the camera to the viewing angle viewed by the user through the screen using the conversion matrix M according to the following equation. Can be.

(Mathematical formula)

ratio: screen height / screen width

far: Forward near-field constant of the camera relative to the camera

near: Forward distance constant of the camera relative to the camera

According to an embodiment, a head mounted display includes: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, wherein the one or more programs are configured to include the head mounted display. Instructions for displaying a plurality of virtual markers on a screen; Instructions for checking whether the virtual markers and a plurality of markers located in front of the head mounted display match on the screen; Calculating an angle between the markers relative to the camera of the head mounted display when the virtual markers and the markers match on the screen; Calculating a distance between the virtual markers displayed on the screen; And a command for calculating a viewing angle viewed by the user through the screen based on an angle between the markers relative to the camera, a distance between the virtual markers, and the size of the screen.

The one or more programs include instructions for calculating a horizontal marker angle or a vertical marker angle between the first marker and the second marker, and calculating a distance between the virtual markers, in a command for calculating an angle between the markers. In FIG. 3, a horizontal virtual marker distance or a vertical virtual marker distance between the first virtual marker and the second virtual marker may be calculated.

The one or more programs may be viewed by the user through the screen based on the horizontal marker angle, the horizontal virtual marker distance, and the width of the screen in a command to calculate a viewing angle that the user looks through the screen. It may include instructions for calculating the viewing horizontal viewing angle.

(Mathematical formula)

The one or more programs may further include instructions for converting the viewing angle of the camera into a viewing angle viewed by the user through the screen using a transformation matrix M according to the following equation.

(Mathematical formula)

ratio: screen height / screen width

far: Forward near-field constant of the camera relative to the camera

near: Forward distance constant of the camera relative to the camera

The one or more programs may be viewed by the user through the screen based on the vertical marker angle, the vertical virtual marker distance, and the height of the screen in a command to calculate a viewing angle that the user looks through the screen. It may include instructions for calculating the viewing vertical viewing angle.

(Mathematical formula)

(Mathematical formula)

ratio: screen height / screen width

far: Forward near-field constant of the camera relative to the camera

near: Forward distance constant of the camera relative to the camera

According to one embodiment, a computing device includes: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein the one or more programs are configured to display a plurality of virtual markers on a screen of a head mounted display. Command for; Instructions for checking whether the virtual markers and a plurality of markers located in front of the head mounted display match on the screen; Calculating an angle between the markers relative to the camera of the head mounted display when the virtual markers and the markers match on the screen; Calculating a distance between the virtual markers displayed on the screen; And a command for calculating a viewing angle viewed by the user through the screen based on an angle between the markers relative to the camera, a distance between the virtual markers, and the size of the screen.

According to an exemplary embodiment of the present invention, the difference between the viewing angle of the user's eyes and the viewing angle of the camera can be easily corrected by calculating a viewing angle viewed by the user through the screen using the marker member and the virtual marker.

1 is a perspective view showing an optical see-through type head mounted display according to an embodiment of the present invention

2 is a view showing a difference in viewing angles between a user's eyes and a camera in an optical see-through type head mounted display.

3 illustrates a marker member according to an embodiment of the present invention.

4 is a flowchart illustrating a calibration method according to an embodiment of the present invention.

FIG. 5 is a view illustrating a state in which a user wears a head mounted display on a head and positions a marker member in front of FIG.

6 is a diagram illustrating a state in which a virtual marker is displayed on a screen according to an embodiment of the present invention.

7 is a view illustrating a state in which virtual markers and markers are matched to a screen according to an embodiment of the present invention.

8 is a diagram illustrating a state of calculating an angle between markers based on a camera according to an embodiment of the present invention.

9 is a diagram showing a projection space by a transformation matrix according to an embodiment of the present invention.

10 is a block diagram illustrating and describing a computing environment including a computing device suitable for use in example embodiments.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to assist in a comprehensive understanding of the methods, devices, and / or systems described herein. However, this is only an example and the present invention is not limited thereto.

In describing the embodiments of the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, terms to be described below are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of a user or an operator. Therefore, the definition should be made based on the contents throughout the specification. The terminology used in the description is for the purpose of describing embodiments of the invention only and should not be limiting. Unless expressly used otherwise, the singular forms “a,” “an,” and “the” include plural forms of meaning. In this description, expressions such as "comprises" or "equipment" are intended to indicate certain features, numbers, steps, actions, elements, portions or combinations thereof, and one or more than those described. It should not be construed to exclude the presence or possibility of other features, numbers, steps, actions, elements, portions or combinations thereof.

In the following description, the terms "transfer", "communication", "transmit", "receive" and other similar meanings of signals or information are not only meant to directly convey the signal or information from one component to another. It also includes passing through other components. In particular, "transmitting" or "sending" a signal or information to a component indicates the final destination of the signal or information and does not mean a direct destination. The same is true for the "reception" of a signal or information. In addition, in this specification, that two or more pieces of data or information are "related" means that if one data (or information) is obtained, at least a portion of the other data (or information) can be obtained based thereon.

Meanwhile, directional terms such as upper side, lower side, one side, the other side, and the like are used in connection with the orientation of the disclosed drawings. Since the components of the embodiments of the present invention can be positioned in various orientations, the directional terminology is used for the purpose of illustration and not limitation.

In addition, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

In the case of augmented reality using optical see-through type Head Mounted Display, in order to match the real world and virtual objects, 1) camera intrinsic parameter correction, 2) user's eyes And correction of the viewing angle between cameras of the head mounted display, and 3) the user's eyes and the physical position of the camera. One embodiment of the present invention relates to a technique for correcting a viewing angle between an eye of a user and a camera of a head mounted display.

1 is a perspective view showing an optical see-through type head mounted display according to an embodiment of the present invention.

Referring to FIG. 1, the head mounted display 100 may include a first screen 102, a second screen 104, a frame 106, a head mount 108, a camera 110, and a calibration module 112. It may include.

The first screen 102 may be provided transparent or translucent or opaque. The first screen 102 may be provided to correspond to one of two eyes of the user (eg, the right eye). The first screen 102 can include a see-through lens.

The second screen 104 may be provided transparent or translucent or opaque. The second screen 104 may be provided in correspondence with the other of the two eyes of the user (eg, the left eye). The second screen 104 may include a see through lens.

The first screen 102 and the second screen 104 may be provided to adjust the transparency. The user can see the real world in front through the first screen 102 and the second screen 104.

The frame 106 may be provided in combination with the first screen 102 and the second screen 104. The frame 106 may be coupled along the rim of the first screen 102 and the second screen 104. The frame 106 may serve to fix the first screen 102 and the second screen 104. The frame 106 may be provided with a part seated on the user's nose or the like so that the head mounted display 100 does not flow down when the head mounted display 100 is worn.

The head mounting portion 108 is a portion for allowing the head mounted display 100 to be mounted on the head portion of the user. The head mounting unit 108 may include a first head mounting unit 108-1 and a second head mounting unit 108-2. The first head mounting portion 108-1 may be provided rearward from one side of the frame 106. The second head mounting unit 108-2 may be provided rearward from the other side of the frame 106. The first head mounting unit 108-1 and the second head mounting unit 108-2 may be provided to surround the head of the user. Although the head mounting unit 108 has been described as including the first head mounting unit 108-1 and the second head mounting unit 108-2, the head mounting unit 108 is not limited thereto. It may be formed integrally with connecting the other side.

The camera 110 may be provided to photograph the periphery of the head mounted display 100. In an exemplary embodiment, the camera 110 may be provided in the frame 106. The camera 110 may be provided between the first screen 102 and the second screen 104 at the top of the frame 106. However, the mounting position of the camera 110 is not limited thereto.

The calibration module 112 may correct a difference in a field of view (FOV) between the user's eyes and the camera 110. The calibration module 112 may be installed in the frame 106, but is not limited thereto. The calibration module 112 may be installed in the head mounting unit 108.

2 is a view illustrating a difference in a viewing angle between a user's eyes and a camera in an optical see-through type head mounted display. Referring to FIG. 2, the viewing angle A of the camera may refer to the viewing angle viewed by the camera 110, and the viewing angle B of the eye may refer to the viewing angle viewed by the user's eyes.

Here, since the user looks forward through the

screens

102 and 104, it is necessary to correct the viewing angle A of the camera based on the viewing angle C that the user views through the

screens

102 and 104. Accordingly, the calibration module 112 calculates the viewing angle C viewed by the user through the

screens

102 and 104, and then views the viewing angle A of the camera through the

screens

102 and 104. C) to correct for differences between viewing angles. Detailed description thereof will be described later.

In the present specification, the module may mean a functional and structural combination of hardware for performing the technical idea of the present invention and software for driving the hardware. For example, the “module” may mean a logical unit of a predetermined code and a hardware resource for performing the predetermined code, and does not necessarily mean a physically connected code or a kind of hardware. .

In the disclosed embodiment, a marker member marked with a plurality of markers may be used to calculate the viewing angle C that the user sees through the

screens

102 and 104.

3 is a view showing a marker member according to an embodiment of the present invention. Referring to FIG. 3, the marker member 150 may include a base portion 151, a fixing portion 153, and a marker 155.

The base unit 151 may serve to support the marker member 150. The base portion 151 may be formed of a flat plate. The fixing part 153 may protrude from one side of the base part 151. For example, the fixing part 153 may protrude downward from the bottom of the base part 151. The fixing part 153 may be a part in which the user grips the marker member 150. Alternatively, the fixing part 153 may be a part mounted on separate equipment.

A plurality of markers 155 may be provided on one surface of the base portion 151. For example, the marker 155 may be attached to one surface of the base portion 151. The marker 155 may include the first marker 155-1 to the fourth marker 155-4. The first marker 155-1 to the fourth marker 155-4 may have a square shape. The first marker 155-1 and the second marker 155-2 may be provided to be symmetrical with respect to the vertical center axis of the base 151 on one surface of the base 151. The third marker 155-3 and the fourth marker 155-4 may be provided to be symmetrical with respect to the vertical central axis of the base 151 at one lower surface of the base 151. In this case, the first marker 155-1, the third marker 155-3, the second marker 155-2, and the fourth marker 155-4 are based on the horizontal center axis of the base unit 151, respectively. It may be provided symmetrically up and down. However, the shape of the marker 155 and the arrangement form between the markers 155 are not limited thereto.

4 is a flowchart illustrating a calibration method according to an embodiment of the present invention. In the illustrated flow chart, the method is divided into a plurality of steps, but at least some of the steps may be performed in a reverse order, in combination with other steps, omitted, divided into substeps, or not shown. One or more steps may be added and performed. Herein, it is assumed that the viewing angles of the user's right eye and the left eye are the same. Hereinafter, the eye of the user may mean the right or left eye of the user, and the screen may mean a screen corresponding to the right or left eye of the user, and for convenience of description, the first screen ( 102).

Referring to FIG. 4, the calibration module 112 checks whether a calibration event occurs (S 101). The calibration event occurs when calibration is required in the head mounted display 100. For example, a user presses a separate button provided on the head mounted display 100 or performs calibration for the head mounted display 100. Can occur when running an application. Alternatively, the calibration event may occur when the camera 110 of the head mounted display 100 recognizes the front marker member 150.

In an exemplary embodiment, as shown in FIG. 5, with the user wearing the head mounted display 100 on the head and positioning the marker member 150 at a predetermined distance in front of the head mounted display 100, By pressing a separate button provided in the head mounted display 100, a calibration event may be generated.

As a result of checking in step S 101, when a calibration event occurs, the calibration module 112 renders and displays a plurality of virtual markers 121 on the screen 102 of the head mounted display 100 (S 103). Here, the virtual marker 121 may be a virtual image formed to correspond to the markers 155 of the marker member 150.

6 is a view showing a state in which the virtual marker 121 is displayed on the screen 102 according to an embodiment of the present invention. Referring to FIG. 6, the virtual marker 121 may include a first virtual marker 121-1 to a fourth virtual marker 121-4. The first virtual marker 121-1 to the fourth virtual marker 121-4 may be formed to correspond to the first marker 155-1 to the fourth marker 155-4, respectively. That is, the first virtual marker 121-1 to the fourth virtual marker 121-4 are formed such that the shape of the first marker 155-1 to the fourth marker 155-4 and the arrangement form between the markers correspond to each other. It may be.

The calibration module 112 may display the virtual marker 121 at the center of the screen 102. In this case, the first virtual marker 121-1, the second virtual marker 121-2, the third virtual marker 121-3, and the fourth virtual marker 121-4 are respectively vertical centers of the screen 102. It may be displayed symmetrically with respect to the axis. In addition, the first virtual marker 121-1, the third virtual marker 121-3, the second virtual marker 121-2, and the fourth virtual marker 121-4 are respectively horizontal centers of the screen 102. It may be displayed symmetrically with respect to the axis.

The virtual marker 121 is projected onto the screen 102 when the user looks at the marker member 150 positioned at a preset reference distance (for example, 1 m) while the user wears the head mounted display 100 on the head. The markers 155 may be formed by modeling the markers 155.

When the virtual markers 121 are displayed on the screen 102, the markers 155 of the marker member 150 located in front of the screen 102 are projected and displayed. That is, since the user sees the marker member 150 located in front of the screen 102, not only the virtual marker 121 is displayed on the screen 102, but the markers 155 are projected and displayed.

Next, the calibration module 112 checks whether the virtual markers 121 and the markers 155 are matched (matched) on the screen 102 (S 105). In an example embodiment, the user may move the marker member 150 on the screen 102 to match the virtual markers 121 and the markers 155. That is, the virtual marker 121 and the markers 155 on the screen 102 may be matched while the marker member 150 is moved back and forth or left and right. The marker member 150 may be moved by the user with one hand, but is not limited thereto and may be moved through separate equipment. FIG. 7 is a view illustrating a state in which virtual markers 121 and markers 155 are matched to the screen 102 according to an embodiment of the present invention.

As a result of checking in step S 105, when the virtual markers 121 and the markers 155 are matched on the screen 102, the calibration module 112 calculates an angle between the markers 155 with respect to the camera 110. (S 107).

8 is a diagram illustrating a state of calculating angles between markers 155 based on the camera 110 according to an embodiment of the present invention. Referring to FIG. 8, when the calibration module 112 matches the virtual markers 121 and the markers 155 on the screen 102, the calibration module 112 may be configured between the camera 110 and the marker member 150. The distance may be obtained and the marker member 150 may be photographed through the camera 110. The calibration module 112 is a marker based on the camera 110 based on the distance between the camera 110 and the marker member 150 and posture information (information related to position and rotation) of each marker 155 in the captured image. The angle between 155 may be calculated. In this case, the calibration module 112 may calculate at least one of the horizontal marker angle θ1 and the vertical marker angle θ2.

Here, the horizontal marker angle θ1 is an angle between the center of the first marker 155-1 and the center of the second marker 155-2 with respect to the camera 110 (or the third marker 155-3). Angle between the center and the center of the fourth marker 155-4). That is, the horizontal marker angle may mean an angle between markers arranged in the horizontal direction with respect to the camera 110 in the marker member 150. In addition, the vertical marker angle θ2 is an angle between the center of the first marker 155-1 and the center of the third marker 155-3 with respect to the camera 110 (or the second marker 155-2). Angle between the center and the center of the fourth marker 155-4). That is, the vertical marker angle may mean an angle between markers disposed in the vertical direction with respect to the camera 110 in the marker member 150.

Next, the calibration module 112 calculates the distance between the virtual markers 121 displayed on the screen 102 (S 109). In an exemplary embodiment, the calibration module 112 may calculate at least one of a horizontal virtual marker distance and a vertical virtual marker distance.

Here, the horizontal virtual marker distance is a distance between the center of the first virtual marker 121-1 and the center of the second virtual marker 121-2 (or the center of the third virtual marker 121-3 and the fourth virtual marker ( 121-4) the distance between the center of the). That is, the horizontal virtual marker distance may mean the distance between the virtual markers arranged in the horizontal direction on the screen 102.

In addition, the vertical virtual marker distance is the distance between the center of the first virtual marker 121-1 and the center of the third virtual marker 121-3 (or the center of the second virtual marker 121-2 and the fourth virtual marker ( 121-4) the distance between the center of the). That is, the vertical virtual marker distance may mean the distance between the virtual markers arranged in the vertical direction on the screen 102. The horizontal virtual marker distance and the vertical virtual marker distance may be expressed in units of pixels.

Next, the calibration module 112 is based on the angle between the markers 155 relative to the camera 110, the distance between the virtual markers 121 displayed on the screen 102, and the size of the screen 102. In operation S111, the viewing angle calculated by the user through the screen is calculated. That is, the calibration module 112 measures the ratio relationship between the angle between the markers 155 with respect to the camera 110 and the distance between the virtual markers 121 displayed on the screen 102 and the size of the screen 102. The viewing angle that the user views through the screen may be calculated by applying the viewing angle that the user views through the screen.

In an exemplary embodiment, if the screen 102 is 1024 pixels wide, the horizontal marker angle is 30 degrees, and the horizontal virtual marker distance is 500 pixels, then x (user) is achieved through a ratio relationship of 30: 500 = x: 1024. Calculates that the horizontal viewing angle viewed through the screen is 61 degrees. Through this, the horizontal viewing angle FoVx that the user views through the screen and the vertical viewing angle FoVy that the user views through the screen may be represented by Equation 1 below.

(Equation 1)

Next, the calibration module 112 converts the viewing angle of the camera 110 to the viewing angle viewed by the user through the screen using the transformation matrix (S 113). The calibration module 112 may convert the viewing angle of the camera 110 into the viewing angle that the user views through the screen by using the transformation matrix M shown in Equation 2 below. In this case, the transformation matrix is calculated using the vertical viewing angle (FoVy) viewed by the user through the screen.

(Equation 2)

Here, the ratio represents the ratio between the width and the height of the screen 102, it may be represented by the height of the screen 102 / the width of the screen 102. far represents the front near constant of the camera 110 based on the camera 110, and near represents the front far constant of the camera 110 based on the camera 110. That is, far is a constant indicating from which point the object in front of the camera 110 will be shown through the camera 110, and near is a distance indicating the object in front of the camera 110 through the camera 110. Is a constant. The height of the screen 102 may correspond to the y-axis, the width of the screen 102 may correspond to the z-axis, and the front of the camera 110 may correspond to the z-axis.

9 is a view showing a projection space by the transformation matrix according to an embodiment of the present invention. Referring to FIG. 9, the virtual plane VP1 by the viewing angle of the camera 110 may have a wider range than the virtual plane VP2 by the viewing angle that the user views through the screen, as shown in FIG. 2. . Here, the virtual plane VP1 and the virtual plane VP2 are located on the same plane.

The calibration module 112 converts the viewing angle of the camera 110 into the viewing angle that the user views through the screen using the transformation matrix according to Equation 2. Then, the virtual plane VP1 based on the viewing angle of the camera 110 may be converted into the virtual plane VP2 based on the viewing angle viewed by the user. Since the ratio in equation (2) is the height (y) of the screen 102 / the width (x) of the screen 102,

The value is used to calculate the converted viewing angle on the x-axis,

The value can be used to calculate the converted viewing angle on the y-axis.

Meanwhile, Equation 2 illustrates a case in which the transformation matrix is calculated using the vertical viewing angle FoVy viewed through the screen, but the transformation matrix M is calculated using the horizontal viewing angle FoVx viewed through the screen. ) Can be calculated, as shown in Equation 3.

(Equation 3)

here,

Value is used to calculate the converted viewing angle on the x-axis,

The value can be used to calculate the converted viewing angle on the y-axis. That is, if only one value of the horizontal viewing angle (FoVx) viewed by the user through the screen and the vertical viewing angle (FoVy) viewed by the user through the screen, the viewing angle of the camera 110 is converted into the viewing angle viewed by the user through the screen. You can do it.

10 is a block diagram illustrating and describing a computing environment 10 that includes a computing device suitable for use in example embodiments. In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

The illustrated computing environment 10 includes a computing device 12. In one embodiment, computing device 12 may be a head mounted display device (eg, head mounted display 100). In addition, computing device 12 may be a device for calibration (eg, calibration module 112).

Computing device 12 includes at least one processor 14, computer readable storage medium 16, and communication bus 18. The processor 14 may cause the computing device 12 to operate according to the example embodiments mentioned above. For example, processor 14 may execute one or more programs stored in computer readable storage medium 16. The one or more programs may include one or more computer executable instructions that, when executed by the processor 14, cause the computing device 12 to perform operations in accordance with an exemplary embodiment. Can be.

Computer readable storage medium 16 is configured to store computer executable instructions or program code, program data and / or other suitable forms of information. The program 20 stored in the computer readable storage medium 16 includes a set of instructions executable by the processor 14. In one embodiment, computer readable storage medium 16 includes memory (volatile memory, such as random access memory, nonvolatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash Memory devices, or any other form of storage medium that is accessible by computing device 12 and capable of storing desired information, or a suitable combination thereof.

The communication bus 18 interconnects various other components of the computing device 12, including the processor 14 and the computer readable storage medium 16.

Computing device 12 may also include one or more input / output interfaces 22 and one or more network communication interfaces 26 that provide an interface for one or more input / output devices 24. The input / output interface 22 and the network communication interface 26 are connected to the communication bus 18. The input / output device 24 may be connected to other components of the computing device 12 via the input / output interface 22. Exemplary input / output devices 24 may include pointing devices (such as a mouse or trackpad), keyboards, touch input devices (such as touchpads or touchscreens), voice or sound input devices, various types of sensor devices, and / or imaging devices. Input devices, and / or output devices such as display devices, printers, speakers, and / or network cards. The example input / output device 24 may be included inside the computing device 12 as one component of the computing device 12, and may be connected to the computing device 12 as a separate device from the computing device 12. It may be.

While the exemplary embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. . Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims below and equivalents thereof.

Claims

One or more processors, and

A method performed in an optical see-through type head mounted display device having a memory for storing one or more programs executed by the one or more processors, the method comprising:

Displaying a plurality of virtual markers on a screen of the head mounted display;

Checking whether the virtual markers and the plurality of markers located in front of the head mounted display match on the screen;

Calculating an angle between the markers based on a camera of the head mounted display when the virtual markers and the markers match on the screen;

Calculating a distance between the virtual markers displayed on the screen; And

Calculating a viewing angle viewed by the user through the screen based on an angle between the markers relative to the camera, a distance between the virtual markers, and the size of the screen.
The method according to claim 1,

The plurality of markers may include a first marker provided on one surface of a base portion positioned in front of the head mounted display, and a second marker spaced apart from the first marker in a horizontal or vertical direction from one surface of the base portion. and,

The virtual marker includes a first virtual marker corresponding to the first marker and a second virtual marker corresponding to the second marker.
The method according to claim 2,

The calculating of the angle between the markers may include calculating a horizontal marker angle or a vertical marker angle between the first marker and the second marker,

The calculating of the distance between the virtual markers may include calculating a horizontal virtual marker distance or a vertical virtual marker distance between the first virtual marker and the second virtual marker.
The method according to claim 3,

Computing the viewing angle that the user sees through the screen,

And calculating a horizontal viewing angle that the user views through the screen based on the horizontal marker angle, the horizontal virtual marker distance, and the width of the screen.
The method according to claim 4,

The horizontal viewing angle FoVx that the user views through the screen is calculated through the following equation.

(Mathematical formula)
The method according to claim 5,

After calculating the horizontal viewing angle FoVx,

And converting the viewing angle of the camera into the viewing angle viewed by the user through the screen by using the transformation matrix M according to the following equation.

(Mathematical formula)

ratio: screen height / screen width

far: Forward near-field constant of the camera relative to the camera

near: Forward distance constant of the camera relative to the camera
The method according to claim 3,

Computing the viewing angle that the user sees through the screen,

Calculating a vertical viewing angle that the user sees through the screen based on the vertical marker angle, the vertical virtual marker distance, and the height of the screen.
The method according to claim 7,

The vertical viewing angle FoVy viewed by the user through the screen is calculated through the following equation.

(Mathematical formula)
The method according to claim 8,

After calculating the vertical viewing angle FoVy,

And converting the viewing angle of the camera into the viewing angle viewed by the user through the screen by using the transformation matrix M according to the following equation.

(Mathematical formula)

ratio: screen height / screen width

far: Forward near-field constant of the camera relative to the camera

near: Forward distance constant of the camera relative to the camera
One or more processors;

Memory; And

An optical see-through type head mounted display comprising one or more programs,

The one or more programs are stored in the memory and configured to be executed by the one or more processors,

The one or more programs,

Instructions for displaying a plurality of virtual markers on a screen of the head mounted display;

Instructions for checking whether the virtual markers and a plurality of markers located in front of the head mounted display match on the screen;

Calculating an angle between the markers relative to the camera of the head mounted display when the virtual markers and the markers match on the screen;

Calculating a distance between the virtual markers displayed on the screen; And

And a command for calculating a viewing angle viewed by the user through the screen based on the angle between the markers relative to the camera, the distance between the virtual markers, and the size of the screen.
The method according to claim 10,

The plurality of markers may include a first marker provided on one surface of a base portion positioned in front of the head mounted display, and a second marker spaced apart from the first marker in a horizontal or vertical direction from one surface of the base portion. and,

The virtual marker includes a first virtual marker corresponding to the first marker and a second virtual marker corresponding to the second marker.
The method according to claim 11,

The one or more programs,

In a command to calculate an angle between the markers, calculating a horizontal marker angle or a vertical marker angle between the first marker and the second marker,

In a command to calculate a distance between the virtual markers, calculating a horizontal virtual marker distance or a vertical virtual marker distance between the first virtual marker and the second virtual marker.
The method according to claim 12,

The one or more programs,

In a command for calculating a viewing angle that the user sees through the screen,

And instructions for calculating a horizontal viewing angle that the user sees through the screen based on the horizontal marker angle, the horizontal virtual marker distance, and the width of the screen.
The method according to claim 13,

The horizontal viewing angle FoVx that the user views through the screen is calculated through the following equation.

(Mathematical formula)
The method according to claim 14,

The one or more programs,

And converting the viewing angle of the camera into a viewing angle viewed by the user through the screen using a conversion matrix M according to the following equation.

(Mathematical formula)

ratio: screen height / screen width

far: Forward near-field constant of the camera relative to the camera

near: Forward distance constant of the camera relative to the camera
The method according to claim 12,

The one or more programs,

In a command for calculating a viewing angle that the user sees through the screen,

And calculating a vertical viewing angle the user views through the screen based on the vertical marker angle, the vertical virtual marker distance, and the height of the screen.
The method according to claim 16,

The vertical viewing angle FoVy viewed by the user through the screen is calculated through the following equation.

(Mathematical formula)
The method according to claim 17,

The one or more programs,

And converting the viewing angle of the camera into a viewing angle viewed by the user through the screen using a conversion matrix M according to the following equation.

(Mathematical formula)

ratio: screen height / screen width

far: Forward near-field constant of the camera relative to the camera

near: Forward distance constant of the camera relative to the camera
One or more processors;

Memory; And

Contains one or more programs,

The one or more programs are stored in the memory and configured to be executed by the one or more processors,

The one or more programs,

Instructions for displaying a plurality of virtual markers on a screen of a head mounted display;

Instructions for checking whether the virtual markers and a plurality of markers located in front of the head mounted display match on the screen;

Calculating an angle between the markers relative to the camera of the head mounted display when the virtual markers and the markers match on the screen;

Calculating a distance between the virtual markers displayed on the screen; And

And calculating a viewing angle the user views through the screen based on an angle between the markers relative to the camera, a distance between the virtual markers, and the size of the screen.