KR102122339B1 - Method for controlling head mounted display, computing device and computer-readable medium - Google Patents

Abstract

The control method of a head mounted display (HMD) includes setting a middle area of a screen and a limit area existing outside the middle area, tracking a user's gaze point, displaying content in the middle area, And displaying the control button in the limit area and controlling the display of the content according to the function set in the control button based on the tracked gaze point.

Description

METHOD FOR CONTROLLING HEAD MOUNTED DISPLAY, COMPUTING DEVICE AND COMPUTER-READABLE MEDIUM

The present invention relates to a head mounted display (HMD) control method, a computing device and a computer readable medium.

A head mounted display (HMD) is a head mounted display, and is a next-generation display device that allows a user to view an image directly in front of the eye through the HMD when the user mounts the HMD on the head. HMD also provides video content to be enjoyed on a large screen, and is applied to various fields, such as medical devices, and is used for surgery or diagnosis. In connection with such HMD, Korean Patent Publication No. 2009-0105485, which is a prior art, discloses a multimedia providing system using HMD and a method for providing the same.

However, the existing HMD had to be input by the user using an external device in order to control the HMD screen. This has caused the user to control the menu or content while viewing the HMD screen. Accordingly, there is a need for a control method that allows a user to operate naturally while viewing an HMD screen.

When a user uses content or an application through the wearing of an HMD, it is intended to provide an HMD control method, a computing device, and a computer-readable medium capable of controlling the screen or menu of the HMD with only the user's gaze movement or head movement. In addition, an object of the present invention is to provide an HMD control method, a computing device, and a computer-readable medium capable of naturally operating a menu selection, scrolling, or the like while a user views the screen through wearing the HMD. However, the technical problems to be achieved by the present embodiment are not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention includes setting an intermediate area of a screen and a limit area existing outside the intermediate area, tracking a user's gaze point, and an intermediate area. It is possible to provide a control method of an HMD including displaying content, displaying a control button in a limit area, and controlling display of the content according to a function set in the control button based on the tracked gaze point.

Further, another embodiment of the present invention includes a memory and a processing unit arranged to interface with the memory, the processing unit sets a middle area of the screen and a limit area existing outside the middle area, and sets the user's gaze point. It is possible to provide a computing device configured to track, display content in an intermediate area, display a control button in a limit area, and control display of content according to a function set in the control button based on the tracked gaze point.

In addition, another embodiment of the present invention, when executed by a computing device, sets the computing device to a middle area of the screen and a limit area existing outside the middle area, tracks the user's gaze point, and provides It is possible to provide a computer-readable medium for displaying content, displaying a control button in a limit area, and controlling display of the content according to a function set in the control button based on the tracked gaze point.

The above-described problem solving means are merely examples, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the above-described problem solving means of the present invention, when the user uses the content or application through the wearing of the HMD, the HMD control capable of controlling the screen or menu of the HMD only by moving the user's gaze or moving the head Methods, computing devices, and computer readable media. In addition, it is possible to provide a HMD control method, a computing device, and a computer-readable medium capable of naturally operating a menu selection, scrolling, or the like while a user views the screen through wearing the HMD.

1 is a configuration diagram of an HMD control system according to an embodiment of the present invention.
2 is a configuration diagram of a computing device according to an embodiment of the present invention.
3A to 3B are diagrams for explaining a process of setting a limit area and an intermediate area according to an embodiment of the present invention.
4A to 4B are diagrams for explaining a process of controlling display of content through a control button according to an embodiment of the present invention.
5A to 5C are diagrams illustrating a process of scrolling content according to an embodiment of the present invention.
6 is a diagram for explaining a process of mixing and using a scroll function and a control button according to an embodiment of the present invention.
7A to 7B are diagrams illustrating menu buttons according to an embodiment of the present invention.
8 is a view showing an input button in an intermediate region according to an embodiment of the present invention.
9A to 9B are diagrams for explaining a process of controlling a user terminal according to another embodiment of the present invention.
10A to 10B are diagrams for explaining a process of controlling a user terminal according to another embodiment of the present invention.
11 is a flowchart illustrating an HMD control method according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains may easily practice. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . Also, when a part is said to "include" a certain component, it means that the component may further include other components, not exclude other components, unless specifically stated otherwise. However, it should be understood that the existence or addition possibilities of numbers, steps, actions, components, parts or combinations thereof are not excluded in advance.

In the present specification, the term “unit” includes a unit realized by hardware, a unit realized by software, and a unit realized by using both. Further, one unit may be realized by using two or more hardware, and two or more units may be realized by one hardware.

Some of the operations or functions described in this specification as being performed by a terminal or device may be performed instead on a server connected to the corresponding terminal or device. Similarly, some of the operations or functions described as being performed by the server may be performed in a terminal or device connected to the corresponding server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an HMD control system 1 according to an embodiment of the present invention. Referring to FIG. 1, the HMD control system 1 may include a server 100 and a user terminal 200. The server 100 and the user terminal 200 illustrated in FIG. 1 exemplarily show components that can be controlled by the HMD control system 1. Each component of the HMD control system 1 of FIG. 1 is generally connected through a network. For example, as illustrated in FIG. 1, the user terminal 200 may be connected to the server 100 through a network. However, if the user terminal owns the content itself, the configuration related to the server 100 and the network may be omitted in this embodiment. In addition, for example, when the user terminal 200 such as a smart glass is connected to another user terminal (not shown) such as a smart phone through Bluetooth, Wi-Fi, etc., the other user terminal (not shown) holds the content. And perform some of the processing process instead.

The network refers to a connection structure capable of exchanging information between each node such as terminals and servers, and examples of such a network include Wi-Fi, Bluetooth, Internet, Local Area Network (LAN), Wireless Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), 3G, 4G, LTE, and the like are included, but are not limited thereto.

The server 100 may receive content from the user terminal 200 and transmit the requested content to the user terminal 200. At this time, the server 100 transmits various types of content such as VOD (Video On Demands), TV program, sound source, E-book, web browser to the user terminal 200 in 2D or 3D type. Can be.

The user terminal 200 may set a middle area of the screen and a limit area existing outside the middle area. At this time, the user terminal 200 may display a dot in the center of the screen. The user terminal 200 may display a horizontal bar on the screen and move the horizontal bar up and down on the screen. In addition, the user terminal 200 may display a vertical bar on the screen and move the vertical bar left and right on the screen. The user terminal 200 may receive user input while the horizontal bar and vertical bar are moving, and set a limit area based on the user input. At this time, the user terminal 200 may set the middle area of the screen at a preset ratio with respect to the limit area. Also, the user terminal 200 may set the middle area of the screen using the same method as the limit area setting method.

The user terminal 200 may track the user's gaze point. At this time, the user terminal 200 may recognize the user's gaze using the built-in camera and track the gaze point. Also, the user terminal 200 may sense the user's head movement and detect the sensed head movement. At this time, the user terminal 200 may detect a user's head movement using various sensors, for example, an acceleration sensor or a gyroscope sensor.

The user terminal 200 may display content in the middle area and control buttons in the limit area. At this time, the user terminal 200 may give a selection effect to the control button when the user's gaze point moves from the middle area to the control button and then back to the middle area. For example, when the user's gaze point moves to the control button, the user terminal 200 may give a touch down effect to the control button. For another example, when the user's gaze point moves from the control button to the middle area, the user terminal 200 may give a touch up effect to the control button.

The user terminal 200 may control the display of the content according to the function set in the control button based on the tracked gaze point of the user. For example, the user terminal 200 may scroll the content when the user's gaze point is moved from the middle area to the limit area. At this time, when the user terminal 200 reaches the end of the content by scrolling the content, the user terminal 200 may indicate that the end of the content is outside the middle area, for example, in the outer area contacting the middle area. In addition, the user terminal 200 may increase the scrolling speed of the content as the user's gaze point moves away from the middle area, and may decrease the scrolling speed of the content before reaching the end of the content by scrolling the content. For another example, the user terminal 200 may change the playback speed of the content when the user's gaze point is moved from the middle area to the limit area. For another example, the user terminal 200 may change the playback volume of the content when the user's gaze point is moved from the middle area to the limit area.

An example of such a user terminal 200 may include a display such as a smart glass (201, Smart Glass), a head mounted display (HMD), or a head up display (HUD) that can be worn on the user's body. . In addition, an example of the user's terminal 200 is a notebook (notebook), a netbook (Netbook), an ultrabook (UltraBook), a subnotebook (SubNotebook), a desknote (DeskNote), UMPC (Ultra-Mobile PC) with a built-in camera ), Desktop, and the like. In addition, examples of the user terminal 200 include PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (Personal Digital Assistant) Guarantees portability and mobility such as International Mobile Telecommunication (2000), Code Division Multiple Access (CDMA)-2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (Wibro) terminal, and smart phone It may be a wireless communication device. In addition, an example of the user terminal 200 may include all types of handheld-based wireless communication devices, such as a smart pad and a tablet PC. Hereinafter, an HMD, which is one example of the user terminal 200, will be described as an example.

2 is a configuration diagram of a computing device according to an embodiment of the present invention. Referring to FIG. 2, the computing device is a device included in the HMD or mounted on the HMD, and may be a computing device that controls a head mounted display (HMD). Alternatively, the computing device may be an external device communicatively connected to the HMD, for example, a smart phone.

The computing device includes a memory 210 and a processing unit 220 arranged to interface with the memory 210. Content may be stored in the memory 210, and the processing unit 220 may set a middle area of the screen and a limit area existing outside the middle area. In addition, when the computing device is an external device communicatively connected to the HMD, a wired or wireless communication unit (not shown) may be additionally provided.

3A to 3B are diagrams for explaining a process of setting a limit area and an intermediate area according to an embodiment of the present invention.

3A is a diagram for explaining a process of setting a limit area. Referring to FIG. 3A, the processing unit 220 displays a dot 310 in the center of the screen. Further, the processing unit 220 may display the horizontal bar 320 on the screen, and move the horizontal bar 320 up and down. Also, the processing unit 220 may display a vertical bar (not shown) on the screen, and move the vertical bar (not shown) left and right. The processing unit 220 may receive user input while the horizontal bar 320 and the vertical bar (not shown) are moving, and set the limit area 330 based on the user input.

For example, while the user gazes at the center point 310 displayed at the center of the screen, the processing unit 220 displays the horizontal bar 320 at the center of the screen, and moves the horizontal bar 320 to the top of the screen I can do it. At this time, the horizontal bar 320 may move slowly at a constant speed, or may skip and move at regular intervals. While the user gazes at the center point 310 and feels that the horizontal bar 320 moving to the top of the screen has reached the limit of the field of view, the user inputs an input to set the upper limit and the processing unit 220 performs the horizontal bar at this time. The position of 320 may be set to the upper end of the limit area.

For example, the upper limit may be set by allowing the user to move the gaze from the center point 310 over a predetermined distance or to select this button by disposing a button around the center point 310. Alternatively, the upper limit may be set by a user nodding the head, a user touching the exterior of the HMD, or a button pressing input.

For another example, while the user gazes at the center point 310 displayed at the center of the screen, the processing unit 220 displays a vertical bar (not shown) at the center of the screen, and the vertical bar (not shown) is displayed on the screen. You can move it to the right. At this time, the vertical bar (not shown) may be moved slowly at a constant speed, or may be skipped at regular intervals. When the user feels that a vertical bar (not shown) moving to the right side of the screen while gazing at the center point 310 has reached the limit of the field of view, the user inputs an input to set the right limit, and the processing unit 220 The position of the vertical bar (not shown) can be set to the right end of the limit area. As such, the processing unit 220 may set the top and bottom of the limit area 330 through the horizontal bar 320 and the right and left sides of the limit area 330 through the vertical bar (not shown). have. At this time, the user may manually change the size of the set limit area 330.

The horizontal bar 320 and the vertical bar (not shown) may be, for example, lines having a constant thickness or strings.

3B is a diagram for explaining a process of setting the intermediate region 370. 3A and 3B, the processing unit 220, the processing unit 220, the intermediate region 370, a preset ratio for the limit region 330, for example, 2/3, 3/4, It can be set to 1/2 or the like. Alternatively, the processing unit 220 displays a horizontal bar 360 on the screen, moves the horizontal bar 360 up and down, displays a vertical bar (not shown) on the screen, and displays a vertical bar (not shown). Moving left and right, the user may directly set the middle area 370 similarly to the method of setting the limit area. Also, the user may manually change the size of the set intermediate region 370.

The processing unit 220 may control the HMD by tracking the gaze point of the user. At this time, the HMD can track the user's gaze point through the built-in camera. Also, the processing unit 220 may sense a user's head movement and control content based on the sensed head movement. At this time, the HMD can detect the user's head movement through a sensor capable of detecting the movement. For example, the processing unit 220 may select a menu in a direction in which the user nods the head through the detected head movement. For another example, when the user slowly raises or lowers the head, the processing unit 220 may enlarge or reduce the content in response to the user's head movement. As another example, when the user moves the head slowly left or right, the processing unit 220 may scroll the content in response to the user's head movement. As another example, when the user's gaze point is located on the control button, the processing unit 220 may detect the nod of the user's head, recognize the selection of the corresponding control button, and execute a function set for the selected control button. . As another example, when the user moves the head left or right, the processing unit 220 may detect the user's head movement and rotate the content or menu left or right.

A sensor capable of detecting motion may include, for example, an acceleration sensor, a gyroscope sensor, and the like.

4A to 4B are diagrams for explaining a process of controlling display of content through a control button 440 according to an embodiment of the present invention.

4A is a diagram showing a control button 440 in a limit area 410 of a screen. The processing unit 220 may display content in the middle region 420 and control buttons in the limit region 410. At this time, the control button may be located on the left and right sides of the limit area 410, such as the menu PP 430 and the menu FW 435. Also, the control buttons may be displayed below the limit area 410, for example, as in the menu 1 441, the menu 2 442, and the menu 3 443.

4B is a diagram for explaining a process of controlling display of content. 4A and 4B, the processing unit 220 may control the display of content according to a function set in the control button 440 based on the tracked gaze point of the user. The arrow in FIG. 4B indicates the gaze point of the user's gaze. For example, when the user gazes at one control button 440 for a predetermined time or longer, the processing unit 220 may execute a function set in the corresponding control button 440. In addition, when the user's gaze point returns from the intermediate region 420 to the intermediate region 420 again through the specific control button 440, the processing unit 220 immediately controls the control button 220 immediately before returning to the intermediate region 420 ( 440) can be executed.

The processing unit 220 provides a selection effect for the control button when the user's gaze point moves from the middle area 420 to the control button of the limit area 410 and then back to the middle area 420. Can be. For example, when the user tries to execute a function set in the control button of'Menu 2'442, the user sets the gaze point to the middle area 420, menu 1 441, menu 2 442, and the middle area. Moving in 420 order, the processing unit 220 may track the gaze point of the moved user. At this time, when the user's gaze point is moved from the middle area 420 to the control button of'Menu 1'441, the processing unit 220 touches down the control button of'Menu 1'441 ( Touch Down) effect. Touch-down generally means that a button is pressed through an input device such as a keyboard, mouse, or touch screen. In addition, when the user's gaze point is moved from the control button of'Menu 1'441 to the control button of'Menu 2'442, the processing unit 220 restores the control button of'Menu 1'441. Returning, and may provide a touch-down effect to the control button of'Menu 2'442. Touch-up generally means releasing an input device on which a button is pressed. The user may move the gaze point from the'menu 2'442 to the middle area 420 to execute the function set in the control button of the'menu 2'442. At this time, when the user's gaze point moves from the control button to the middle area 420, the processing unit 220 may execute the function set in the control button of the'menu 2'442.

For another example, when the user wants to execute the function set in the control button of'Menu 2'442, the user moves the gaze point directly from the middle area 420 to the control button of'Menu 2'442, , May move back to the middle area 420 in the'menu 2'442, and the processing unit 220 may track the gaze point of the moved user. When the user's gaze point is moved to the control button of the'menu 2'442, the processing unit 220 gives a touch down effect to the control button of the'menu 2'442, and the user's gaze point is intermediate again When moving to the area 420, the function set in the control button of the'menu 2'442 may be executed.

As another example, when the user wants to execute the function set in the control button of'Menu 2'442, the user moves the gaze point directly from the middle area 420 to the control button of'Menu 2'442. Can be. At this time, when the user gazes at'Menu 2'442 for a predetermined time or longer, the processing unit 220 may execute a function set in the control button of'Menu 2'4420.

For another example, when the user detects the user's head movement, such as nodding his head while staring at the control button of'Menu 2'442, the function set in the control button of Menu 2 is executed. I can do it.

In the above examples, in addition to the touch-down effect on the control button of the limit area 410, the current gaze point may be expressed by displaying a mouse cursor as in some computers.

5A to 5C are diagrams illustrating a process of scrolling content according to an embodiment of the present invention.

5A is a view showing an intermediate region 510 and a limiting region 550. Referring to FIG. 5A, the processing unit 220 may scroll 520 the content displayed in the middle area 510 when the user's gaze point moves from the middle area 510 to the limit area 550. For example, the processing unit 220 may scroll the content to the left when the user's gaze point is moved from the middle area 510 to the left limit area 550, and the user's gaze point is the middle area 510. ), the content may be scrolled to the right when moving to the right limit area 550. At this time, the processing unit 220 may increase the speed of the scrolling 520 of the content as the user's gaze point is farther from the middle area 510, and the content before reaching the end of the content by the scrolling 520 of the content The speed of the scroll 520 can be reduced. At this time, a scroll effect such as a triangle or an arrow may be additionally expressed in the limit area 550 or the middle area 510 during scrolling. In addition, the size of the figure can be adjusted or animated according to the speed.

For another example, the processing unit 220 may change the playback speed of the content when the user's gaze point is moved from the middle area 510 to the limit area 550. For example, when the user's gaze point moves from the middle area 510 to the left limit area 550, the processing unit 220 may decelerate the playback speed of the content, and the user's gaze point is the middle area ( When moving from 510 to the right limit area 550, the playback speed of the content can be accelerated.

For another example, the processing unit 220 may change the playback volume of the content when the user's gaze point is moved from the middle area 510 to the limit area 550. For example, when the user's gaze point moves from the middle area 510 to the left limit area 550, the processing unit 220 may reduce the playback volume of the content, and the user's gaze point is the middle area ( When moving from 510 to the right limit region 550, the volume of reproduction of the content may be increased.

5B is a diagram for explaining a process in which the reproduction speed of content is accelerated. 5A and 5B, a sensor built into the HMD senses a user's head movement, and the processing unit 220 can control content based on the sensed head movement. The processing unit 220 may scroll the content 520 when the user's gaze point is moved from the middle area 510 to the limit area 550. At this time, as the user's gaze point moves away from the middle area 510, the scrolling speed of the content may be accelerated 530. Alternatively, when the user's head movement moves in the same direction as the gaze point, the processing unit 220 may further accelerate the scroll speed 530.

5C is a diagram showing that the end of content has been reached by scrolling. Referring to FIG. 5C, when the end of the content is reached by scrolling the content, the processing unit 220 indicates (540) that the content is at the end of the content outside the middle area, for example, in an outer area contacting the middle area. Can give effect. Alternatively, when the end of the content is reached by the scrolling of the content, the processing unit 220 may provide an effect of displaying (not shown) the end of the content outside the middle area, for example, in an outer area contacting the middle area. Can be. At this time, the processing unit 220 may give a predetermined indication to display 540 the end of the content, and may return the end of the scrolled content to a position before scrolling. Or, the processing unit 220 may not move the end of the content.

6 is a diagram for explaining a process of mixing and using a scroll function and a control button according to an embodiment of the present invention. Referring to FIG. 6, the processing unit 220 may control content by mixing and using a scroll function and a control button from a user's gaze point. For example, suppose that the user reads one page of an e-book using the scroll function and moves from the current page to the next page. At this time, if the user's gaze point moves from the middle area 610 to the limit area 600, the processing unit 220 may perform a scroll function on the current page of the e-book. In addition, when the user's gaze point moves to the control button 620, the processing unit 220 may impart a touch down effect to the corresponding control button 620. When the user's gaze point returns to the intermediate region 610 again through the limit region 600, the processing unit 220 may perform a function set in the control button 620. At this time, the function set in the control button 620 may be a “page turning” function.

7A to 7B are diagrams showing menu buttons in an intermediate area according to an embodiment of the present invention. The processing unit 220 may display some of the menu buttons 720 to 740 in the middle area 710 and execute a function set in the menu buttons 720 to 740 based on the tracked gaze point of the user.

7A is a diagram in which the sub-menu button 730 is displayed in the middle area 710. Referring to FIG. 7A, when the user's gaze point is moved from the middle area 710 to the'menu 1'721, the processing unit 220 may display the'submenu 1'corresponding to the'menu 1'721 ( 731) and'submenu 2'732 may be displayed on the intermediate region 710. For example, the user's gaze point is moved in the order of the middle area 710,'menu 1'721,'sub menu 1'731,'sub menu 2'732, and the middle area 710. In case, the processing unit 220 recognizes the selection of the'sub-menu 2'732, which is the menu just before the user's gaze point moves to the middle area 710, and performs the function set in the'sub-menu 2'732. It can be done. At this time, when the user's gaze point is moved from the middle area 710 to the'submenu 1'731, the processing unit 220 may give a touchdown effect to the'submenu 1'731. . In addition, when the user's gaze point is moved from the'sub-menu 1'731 to the'sub-menu 2'732, the processing unit 220 returns the original to the'sub-menu 1 731,' A touch down effect may be applied to 2'732. In addition, when the user's gaze point is moved from the'sub-menu 2'732 to the'menu 2'722, the display of the'sub-menu 2'732 may be removed.

7B is a diagram in which a pop-up menu 740 is displayed in the middle area 710. Referring to FIG. 7B, the processing unit 220 may display a pop-up menu 740 in the middle area 710. For example, the user's gaze point is the middle area 710, the pop-up menu 740, the'select 1'menu button 741, the'select 2'menu button 742, the pop-up menu 740, the middle area 710 ), the processing unit 220 recognizes the selection of the'selection 2'menu button 742, which is the menu just before the user's gaze point moves to the middle area 710, and selects the'selection 2'menu. The function set in the button 742 may be performed. Alternatively, when the user's gaze point is stopped for a predetermined time at the'Selection 2'menu button 742, the processing unit 220 recognizes the selection of the'Selection 2'menu button 742, and selects the'Selection 2'menu. The function set in the button 742 may be performed. Or, if the user's gaze point is located at the'selection 2'menu button 742 and the user's head movement is detected, the processing unit 220 recognizes the selection of the'selection 2'menu button 742, and The function set in the selection 2'menu button 742 may be performed.

8 is a view showing an input button in an intermediate region according to an embodiment of the present invention. Referring to FIG. 8, the processing unit 220 may display a plurality of input buttons 821 to 822 in the middle region 810. For example, when the user's gaze point moves in the order of the middle area 810 and the input button 820, the processing unit 220 recognizes the selection of the input button 820 from the user's gaze point, and the input button The function set in 820 may be performed immediately. Alternatively, when the user's gaze point is stopped for a predetermined time on the input button 820, the processing unit 220 may perform a repetitive input operation on the function set in the input button 820. The input button 820 may be used when it is necessary to perform a fast operation such as a game and key typing.

9A to 9C are diagrams for explaining a process of controlling a user terminal according to another embodiment of the present invention. Referring to FIG. 9A, the user terminal 900 may include a camera 910 and a full screen 920. The user may set the entire screen 920 of the user terminal 900 as the entire middle area. Referring to FIG. 9B, the user may set a part of the full screen 920 of the user terminal 900 as the middle area 920 and the rest of the full screen 920 as the limit area 930. The camera 910 may recognize the user's gaze point. At this time, the processing unit 220 may set a function to operate in the upper/lower/left/right or limit area 930 of the intermediate area 920 from the user's gaze point. The function to operate in the middle area 920 or the limit area 930 may include, for example, scrolling, page turning, previous screen, menu selection, and the like. Referring to FIG. 9C, a menu button is set at the bottom of the middle area 920. At this time, when the user's gaze point is moved in the order of the middle area 920,'menu 1'941,'menu 2'942,'menu 3'943, and the middle area 920, processing The unit 220 may recognize the menu selection of the'menu 3'943, which is the menu just before moving to the middle area 920, and perform a function set in the'menu 3'943.

The HMD includes a computer readable medium, which, when executed by a computing device, sets the intermediate area of the screen and a limiting area outside the intermediate area, and tracks the user's gaze point when executed by the computing device. And, it may include a sequence of instructions to display the content in the middle area, display the control button in the limit area, and control the display of the content according to the function set in the control button based on the tracked gaze point.

The HMD also includes a sequence of instructions that, when executed by the computing device, causes the computing device to impart a selection effect on the control button when the gaze point moves from the middle area to the control button and then back to the middle area. Can be.

10A to 10B are diagrams for explaining a process of controlling a user terminal 1000 according to another embodiment of the present invention. Referring to FIG. 10A, the user terminal 1000 may include a camera 1010, and an app window 1030 may be displayed on the entire screen 1020. The menu selection area 1040 may be applied to the app window 1030. For example, when the user terminal 1000 is an operating system (OS) that provides a multi-window, such a menu selection area 1040 may be provided. The menu selection area 1040 may be displayed overlaid on the app window 1030. The menu selection area 1040 may be larger than the size of the app window 1030, or may be the same size as the app window 1030.

10B is a diagram illustrating a process of selecting the menu 1060 through the menu selection area 1040. Referring to FIG. 10B, the processing unit 220 may display a plurality of menus 1061 to 1062 in the menu selection area 1040. The plurality of menus 1061 to 1062 may be set in advance, and may be displayed overlaid on the menu selection area 1040. For example, when the user's gaze point is moved in the order of the app window 1030, the'menu 1'button 1061, and the app window 1030, the processing unit 220 displays the user's gaze point at the app window 1030. It is possible to recognize the selection of the'menu 1'button 1061, which is the menu just before moving to, and perform a function set in the'menu 1'button 1061. At this time, the processing unit 220 may recognize the user's changed gaze point by moving the mouse cursor, such as a mouse move, by tracking the user's gaze point. For another example, when the user's gaze point is moved in the order of the app window 1030, the'menu 1'1061, the'menu 2'1062, and the app window 1030, the processing unit 220 is the user The gaze point of may recognize the selection of the'menu 2'button 1062, which is the menu just before moving to the app window 1030, and may perform a function set in the'menu 2'button 1062.

In addition, when the user gazes at the menu selection area 1040 and then gazes at the entire screen 1020 located outside the menu selection area 1040, the selected menu may be canceled.

11 is a flowchart illustrating an HMD control method according to an embodiment of the present invention. The HMD control method performed by the user terminal 200 according to the embodiment shown in FIG. 11 includes steps performed in time series in the HMD control system 1 according to the embodiment shown in FIG. 1. Therefore, even if omitted below, the content already described with respect to the HMD control system 1 according to the embodiment shown in FIG. 1 is HMD control performed by the user terminal 200 according to the embodiment shown in FIG. 11 Method also applies. Hereinafter, HMD as the user terminal 200 will be described as an example.

In step S1110, the HMD sets a middle area of the screen and a limit area existing outside the middle area. At this time, the middle region may be set at a preset ratio with respect to the limit region. In step S1120, the HMD tracks the user's gaze point. In step S1130, the HMD displays the content in the middle area and the control button in the limit area. In step S1140, the HMD controls the display of the content according to the function set in the control button based on the tracked gaze point.

Although not shown in FIG. 11, the step of setting the limit area is for displaying a horizontal bar on the screen and moving the horizontal bar up and down, displaying a vertical bar on the screen and moving the vertical bar left and right, and moving the horizontal bar and vertical bar. Accordingly, the method may include receiving a user input and setting a limit area based on the user input. At this time, the step of setting the limit area may further include displaying a dot at the center of the screen.

Although not shown in FIG. 11, the step of controlling the display of content may include a step of giving a selection effect to the control button when the gaze point moves from the middle area to the control button and then back to the middle area. have.

Although not shown in FIG. 11, the step of controlling the display of the content is when the gaze point is moved to the control button, the step of giving a touch down effect to the control button and the gaze point is moved from the control button to the middle area In one case, it may include the step of imparting a touch up effect to the control button.

Although not illustrated in FIG. 11, controlling the display of the content may include scrolling the content when the gaze point moves from the middle area to the limit area. At this time, the step of scrolling the content includes providing an effect of displaying the end of the content outside the middle area, for example, outside the middle area when the end of the content is reached by scrolling the content. can do. In addition, scrolling the content may include increasing the scrolling speed of the content as the gaze point moves away from the middle area. Further, scrolling the content may include reducing the scrolling speed of the content before reaching the end of the content by scrolling the content.

Although not shown in FIG. 11, controlling the display of the content may include changing the reproduction speed of the content when the gaze point moves from the middle area to the limit area.

Although not shown in FIG. 11, controlling the display of the content may include changing the playback volume of the content when the gaze point moves from the middle area to the limit area.

Although not illustrated in FIG. 11, the method may further include sensing a user's head movement and controlling content based on the sensed head movement.

Although not illustrated in FIG. 11, the method may further include displaying a menu button in the middle region and executing a function set in the menu button based on the tracked gaze point.

In the above description, steps S1110 to S1140 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. Also, some steps may be omitted if necessary, and the order between the steps may be changed.

The HMD control method described with reference to FIG. 11 may also be implemented in the form of a recording medium including instructions executable by a computer, such as program modules executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism, and includes any information delivery media.

The above description of the present invention is for illustration only, and a person having ordinary knowledge in the technical field to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and it should be interpreted that all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof are included in the scope of the present invention. do.

100: server
200: user terminal
210: memory
220: processing unit

Claims (20)

In the control method of the head mounted display (HMD: Head Mounted Display)
Setting an intermediate area of the screen and a limit area existing outside the intermediate area;
Tracking a user's gaze point;
Displaying content in the middle area and displaying a control button in the limit area; And
Controlling the display of the content according to the function set in the control button based on the tracked gaze point
Including,
The step of setting the limit area
Displaying a horizontal bar on the screen and moving the horizontal bar up and down;
Displaying a vertical bar on the screen and moving the vertical bar left and right;
Receiving a user input according to the movement of the horizontal bar and the vertical bar; And
Setting the limit area based on the user input
Including, HMD control method.
According to claim 1,
Controlling the display of the content is
And when the gaze point is moved from the intermediate region to the control button and then to the intermediate region, adding a selection effect to the control button.
According to claim 1,
Controlling the display of the content is
When the gaze point moves to the control button, applying a touch down effect to the control button; And
When the gaze point is moved from the control button to the intermediate region, a step of giving a touch up effect to the control button
Including, HMD control method.
According to claim 1,
Controlling the display of the content is
Scrolling the content when the gaze point moves from the intermediate area to the limit area
Including, HMD control method.
The method of claim 4,
The step of scrolling the content
When the end of the content is reached by scrolling the content, adding an effect of displaying the end of the content to an outer area of the middle area
Including, HMD control method.
The method of claim 4,
The step of scrolling the content
Increasing the scrolling speed of the content as the gaze point is further away from the intermediate region
Including, HMD control method.
The method of claim 4,
The step of scrolling the content
Decreasing the scrolling speed of the content before reaching the end of the content by scrolling the content
Including, HMD control method.
According to claim 1,
Controlling the display of the content is
Changing the reproduction speed of the content when the gaze point moves from the intermediate region to the limit region.
Including, HMD control method.
According to claim 1,
Controlling the display of the content is
Changing the reproduction volume of the content when the gaze point moves from the intermediate region to the limit region
Including, HMD control method.
delete According to claim 1,
The step of setting the limit area
Displaying a dot in the center of the screen
Further comprising, HMD control method.
According to claim 1,
The intermediate region is set at a preset ratio with respect to the limit region, HMD control method.
According to claim 1,
Sensing the user's head movement; And
Controlling the content based on the sensed head movement
Further comprising, HMD control method.
According to claim 1,
Displaying a menu button in the middle area; And
Executing a function set in the menu button based on the tracked gaze point
Further comprising, HMD control method.
In the computing device for controlling the head mounted display (HMD),
Memory, and
Processing unit arranged to interface with the memory
Including,
The processing unit,
Set an intermediate area of the screen and a limit area existing outside the intermediate area,
Track the user's gaze,
Content is displayed in the middle area, a control button is displayed in the limit area,
Based on the tracked gaze point, it is configured to control the display of the content according to the function set in the control button,
The processing unit,
Display a horizontal bar on the screen and move the horizontal bar up and down,
Display a vertical bar on the screen and move the vertical bar to the left and right,
Receive user input according to the movement of the horizontal bar and the vertical bar,
And set the limit area based on the user input.
The method of claim 15,
The processing unit,
And when the gaze point moves from the intermediate area to the control button and then to the intermediate area, the computing device is configured to impart a selection effect to the control button.
The method of claim 15,
The processing unit,
And wherein the gaze point is configured to change the playback speed of the content when the gaze point moves from the intermediate region to the limit region.
delete A computer readable medium comprising a sequence of instructions for controlling a head mounted display (HMD), comprising:
When executed by a computing device, the computing device,
Set an intermediate area of the screen and a limit area existing outside the intermediate area,
Track the user's gaze,
Content is displayed in the middle area, a control button is displayed in the limit area,
Based on the tracked gaze point, to control the display of the content according to the function set in the control button,
Display a horizontal bar on the screen and move the horizontal bar up and down,
Display a vertical bar on the screen and move the vertical bar to the left and right,
Receive user input according to the movement of the horizontal bar and the vertical bar,
And a sequence of instructions configured to set the limit area based on the user input.
The method of claim 19,
When executed by the computing device, instructions that cause the computing device to impart a selection effect to the control button when the gaze point moves from the intermediate area to the control button and then back to the intermediate area A computer readable medium further comprising a sequence.

Images