KR20160118568A - Method and apparatus for providing information terminal with hmd - Google Patents

Abstract

The present invention relates to an information providing service method for a terminal having a head mounted display (HMD), which includes the steps of: displaying a predetermined menu in accordance with an event detected through a user interrupt on an HMD viewing area of a first user; broadcasting contents to be displayed on HMDs of other paired users by activating an item for supporting a screen sharing service between at least one or more other users from the displayed menu; and controlling the contents that are broadcasted from the HMD of the first user and are shared with at least one or more HMDs through an interaction between the users of the HMDs on a viewing area of the displayed screen. The present invention may provide an augmented reality service in addition to a screen to be shared to a plurality of users, and may perform display mirroring and multimedia sharing in a bidirectional scheme.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and an apparatus for providing information of a terminal equipped with an HMD,

The present invention relates to a screen sharing display service between HMD users.

Conventional head-mounted devices or head-mounted displays (HMDs) are mainly used to transmit additional information to a user through a see-through information display device. have.

Recently, efforts are being made to apply an Augmented Reality (AR) service in which information is superimposed on an actual image using a wearable device or HMD in the form of a spectacle.

Augmented reality (AR) is a computer graphics technique that combines virtual objects or information into a real environment to make them look like objects in the original environment.

Due to the spread of the HMD based on the augmented reality, the HMD users want to share the experience more effectively and efficiently, so as to pursue more efficient use of time and convenience.

Therefore, the present invention is applicable to a screen sharing display service that is displayed on at least one or more other HMDs through mirroring of screen data, which is currently visually observed, We want to support the shared screen control service that reflects the HMD user-specific interaction on the screen.

According to an aspect of the present invention, there is provided an information providing service method of a terminal equipped with an HMD (Head Mounted Display), the method comprising the steps of: The method comprising the steps of: broadcasting an item for supporting a screen sharing service among at least one or more other users from the displayed menu to display content to be displayed in a pairing HMD of another user; The method comprising the steps of: controlling, through interaction with each of the HMD users, a viewing area of a screen broadcasted from one user's HMD and shared by at least one or more HMDs.

According to another aspect of the present invention, there is provided an information providing service apparatus of a terminal equipped with an HMD (Head Mounted Display), comprising: a sensor unit and a photographing unit; And displaying a predetermined menu corresponding to an event detected through a user interrupt in the touch screen and displaying at least one screen sharing service among the other users from the displayed menu, When the item to be supported is activated, the content to be displayed is broadcast in the HMD of another paired user, and the content to be broadcast is broadcast and displayed in a viewing area of at least two HMDs, A control unit for controlling the display unit to be controlled through the HMD, Is telephone service-specific items is a list which is characterized in that it comprises a processing unit for processing a predetermined menu menu to be displayed overlaid (overlay) on the screen in which the selected viewing content from the user.

The present invention provides an information providing service of a terminal equipped with an HMD, in which an individual HMD first user provides a detailed enhancement service in addition to a screen to be shared with at least two or more users including second and third users, The display mirroring and the multimedia sharing can be performed in both directions so that the user interaction based on the user gesture can be performed.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an overall flowchart of an information providing service method of a terminal equipped with an HMD according to an embodiment of the present invention; FIG.
BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an information providing service method of a terminal equipped with an HMD, and more particularly,
3 is a diagram illustrating an example of a screen sharing service in an information providing service method of a terminal equipped with an HMD according to an embodiment of the present invention.
FIG. 4 is a schematic conceptualization of experience sharing through screen sharing among HMD users in an information providing service apparatus of a terminal equipped with an HMD according to an embodiment of the present invention; FIG.
5 is a detailed block diagram of an information providing service apparatus of a terminal equipped with an HMD according to an embodiment of the present invention;
6 is a detailed block diagram illustrating an HMD interface configuration for an operation related to a 2D map view-based augmented reality information providing method in an information providing service method of a terminal equipped with an HMD according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It will be appreciated that those skilled in the art will readily observe that certain changes in form and detail may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims. To those of ordinary skill in the art.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a head mount display (HMD), and more particularly, to a head mount display (HMD), in which a currently visually observed current image or content displayed in a separate HMD and corresponding additional information is mirrored In order to support a shared screen display service displayed on at least one or more other HMDs and a shared screen control service in which an HMD user-specific interaction is reflected on a shared screen, a predetermined menu is displayed on the HMD of the first user, Displaying the content overlaid on a currently displayed viewing area, activating an item for supporting a screen sharing service among at least one or more other users from the menu, and displaying the content to be displayed on the paired HMD of another user Bro And the contents shared by at least one or more HMDs are controlled through HMD user-specific interactions, thereby allowing each first user to share with at least two or more users including the second and third users, We intend to provide technologies capable of performing display mirroring and multimedia sharing in both directions to provide precise reinforcement services and to perform user interaction based on mutual user gestures.

Hereinafter, an information providing service method of a terminal equipped with an HMD according to an embodiment of the present invention will be described in detail with reference to FIG. 1 to FIG.

1 is a flowchart illustrating an information providing service method of a terminal equipped with an HMD according to an embodiment of the present invention.

Referring to FIG. 1, in step 110, a predetermined content selected from a user is displayed on a display screen of a wearable terminal including a head wear display or a head wear display.

Here, the content includes a multimedia service that is served over the Internet, or a current image that is currently visually observed by the HMD user. In this case, the current image is an augmented reality service in which information is superimposed on an actual image, , The position is confirmed through a sensor such as GPS, gyro, acceleration, compass, etc. provided in the HMD, and the position information is indirectly measured through the camera using the position as a starting point and supported through a service server linked through a network And the surrounding information is displayed in a superimposed manner.

In addition, the content includes an image input through a camera, a map image, an actual three-dimensional spatial image based on a gaze path of an HMD user, and augmented reality information superimposed on each of the images.

In step 112, it is checked whether or not a user interrupt is generated in the HMD viewing area of the first user. If a user interrupt is generated as a result of the check, a step 114 is performed to display a predetermined menu.

Here, the viewing area refers to an area where a content is displayed on the display screen of the HMD, and the area includes all the screen display services that are visually provided to the user through the HMD, a multimedia service via the Internet, The image information which is currently visually observed or inputted according to the movement of the user's gaze is displayed.

In addition, the user interrupt refers to an intention that a user intends to input through an input unit called a touch screen (or a touch pad) in order to select or activate an object, , Pointing by touch.

In addition, the predetermined menu is overlaid on a screen in which content items supported by the HMD are listed and the content selected by the user is displayed. 2 is a diagram illustrating a display operation in a viewing area of an HMD in an information providing service method of a terminal equipped with an HMD according to an embodiment of the present invention. As shown in FIG. 2, the applied menu is composed of a list of items in a specific service function supported and executed by the HMD, and the predetermined menu 214 includes user interrupts generated in the viewing area 212 in which the current contents are displayed The content is superimposed and displayed on the content.

Referring again to FIG. 1, in step 116, it is checked whether or not an item is activated to support at least one screen sharing service among other users from the displayed menu. If an item for supporting the screen sharing service is activated, And establishes a link for communication connection between the HMDs of other users paired with each other.

More specifically, when a predetermined user, for example, a first user wants to share a screen displayed in a viewing area of his / her HMD with another user, it interrupts a predetermined position (or a preset position) in the viewing area, And activates an item for supporting a screen sharing service among at least one or more other users in the displayed menu through an interrupt.

Referring to FIG. 2 again, the operation of activating the menu items may be realized by selecting a screen sharing item through a user interrupt from a plurality of menu items 216 provided in a predetermined menu, And a lower sub-item included in the screen sharing item is displayed in a window 218 form.

The HMD list of at least two other users that have been previously set and paired is displayed in the lower sub-item, and a selection input for a specific HMD to share the content displayed in the current viewing area is awaited. Thereafter, when at least one HMD is selected in the lower sub-item, the HMD of the first user performs a connection for screen sharing through a communication link request to the corresponding HMD. In the window 218 in which the lower sub-items are displayed, the HMDs whose security policy based authentication has been completed are listed, and a link for communication is established directly without a separate authentication procedure through user interrupts.

Meanwhile, in step 120 of FIG. 1, the content to be displayed is unicasted in the paired HMD of the other user. And activates a screen sharing service item of a predetermined menu displayed in the viewing area from the HMD of the first user to generate unicasting and transmits the unicasting to the HMD terminal to which the receiving party wants to share the screen.

In operation 122, the contents unicasted from the HMD of the first user and shared by at least one HMD are controlled through interaction with the HMD user in the viewing area of the screen on which the contents are displayed.

3 is a view illustrating a screen sharing service in a method of providing information of a terminal equipped with an HMD according to an embodiment of the present invention. Referring to FIG. 3, One-to-one pairing is performed between the HMD and the second HMD to divide the viewing area 311 of each HMD to share the contents services displayed in the viewing area of the HMD with each other, The first position 310 and the second position 312 are divided and displayed in the HMD viewing area.

3, the viewing area 311 of the HMD to which the present invention is applied has a plurality of divided areas 310 and 312, an item corresponding to each of the divided areas 310 and 312 is formed , A first location (310) in the viewing area is provided with a first service associated with the item formed in the divided area, a second location (312) is associated with the item formed in the divided area, Service and a second service are provided.

Wherein the item is a content displayed in a viewing area of a first user or an HMD of another user, the first service is a service in which an actual three-dimensional space inputted through a camera provided in a HMD of a first user is displayed, The service includes a service for displaying an actual three-dimensional space inputted through a camera provided in the HMD of another user that is previously paired with the HMD of the first user.

Here, the viewing area 311 shown in FIG. 3 is for explaining that the screen is divided by taking the first and second services and the first and second positions as an example, and the viewing area screen division in the present invention is not limited to this It is to be noted that the position of each divided screen related to the screen division of the viewing area can be further increased by user selection.

The first and second services are performed by mirroring the screen data of the HMD of another user paired according to the HMD of the first user or establishing the communication connection with the HMD of the first user.

The mirroring is an operation configured to output screen data corresponding to a screen displayed in a viewing area through a screen sharing service supported by the HMD to another HMD having a paired HMD out function. In the HMD to which the present invention is applied, The occurrence of the set event is detected and the mirroring restriction is determined according to the detected event type so as to prevent the unintended information from being exposed to the other person.

The detected event type is set in the HMD to limit the mirroring of the lock screen, the character input device, or the notification. For example, the detected event type may be a gesture of the user, such as covering the detection sensor of the HMD viewing area with a hand gesture).

The items formed in the first and second positions 310 and 312 can be moved in the vertical and horizontal directions through user interrupts.

Meanwhile, FIG. 6 illustrates an HMD interface for a 2D map-based augmented reality information providing operation in an information providing service method of a terminal equipped with an HMD according to an embodiment of the present invention. Referring to FIG. 6, , A 2D map view based on a user-specific location information acquired via GPS is displayed on the viewing area 610 of the user HMD interface by being deformed or superimposed on the AR view.

The 2D map view is performed through activation of an item for which an annotation is supported from a predetermined menu. The 2D map view can be displayed by using a GPS, a gyroscope, an accelerometer, a compass, and a timer provided in the sensor unit 615 Acquires the location information of the user, measures the direction information of the user through the movement of the HMD, and provides the annotation information about the main points of interest included in the image observed in the current HMD through the location annotation 614 And displays it in the corresponding viewing area 610.

The augmented reality view (AR View) is displayed through a content unit 616 that converts a Picture, Video, 3D model, and Voice into meta data and then generates the augmented annotation, and the digital information of the virtual space may be further included have.

The HMD interface of the other user including the corresponding 612 viewing area is also the same as the configuration of the user HMD interface including the above-mentioned 610 viewing area.

According to this configuration, in the HMD to which the present invention is applied, information about a point of interest (or a point at which an annotation is set) located in a current image or a 2D map-view based image positioned in a direction of a user's gaze is superimposed And provides information on the augmented reality information. Further, in providing the augmented reality information, information on different points of interest (or points on which the annotations are set) provided from the user HMD interface and other user interfaces are compared and integrated, To provide excellent and accurate augmented reality information.

Accordingly, the 2D map-view-based AR view displayed in the viewing area 610 of the user HMD interface including the viewing area of 610 is collected from the HMD of at least one other paired user and the HMD viewing Area.

In the foregoing, a method of providing information of a terminal equipped with an HMD according to an embodiment of the present invention has been described.

Hereinafter, an information providing service apparatus of a terminal equipped with an HMD according to an embodiment of the present invention will be described with reference to FIGS.

4 is an exemplary diagram schematically conceptualizing experience sharing through screen sharing among HMD users in an information providing service apparatus of a terminal equipped with an HMD according to an embodiment of the present invention.

4, in the HMD 412 of the first user 40 who wants to share his or her experience with another user HMD, If the touch pad is touched, a predetermined menu is displayed in the viewing area 410, and an item for supporting a screen sharing service among other users set in the preset menu is activated through an interrupt of the first user 40, The content displayed in the viewing area 410 is displayed in the viewing area 414 of the HMD 416 of the other user 41 to thereby display a screen based visual sense shared between the first user and the second user. ), And auditory sense based mutual psychological relationship.

5 is a detailed block diagram of an information providing service apparatus of a terminal equipped with an HMD according to an embodiment of the present invention.

5, the apparatus 500 to which the present invention is applied includes a photographing unit (camera) 510, a touch screen (display unit) 512, a search unit 514, a menu processing unit 516, a control unit 518, A content unit 520, a content unit 522,

The sensor unit 520 includes a GPS, a gyroscope, an accelerometer, a compass, and a timer. The sensor unit 520 acquires positional information of a user using the GPS, and compares the gaze direction information And outputs the measurement result to the controller 518.

The content unit 522 stores contents output through the display unit 512 in the information providing service apparatus 500 of the terminal equipped with the HMD and controls the display unit 512 under the control of the user selection based control unit 518. [ .

The search unit 514 searches neighboring HMDs to share a screen and outputs the neighboring HMDs to the controller 518.

The display unit 512 has a plurality of divided areas, and each divided area is a touch screen in which a viewing area in which a corresponding item is formed is displayed.

The controller 518 controls the overall operation of the information providing service apparatus 500 of the terminal equipped with the HMD.

In addition, the controller 518 displays a predetermined menu corresponding to the detected event through the user interface of the touch screen 512, and displays an item for supporting at least one screen sharing service among other users from the displayed menu (Or broad) casts the content to be displayed in the pairing HMD of another user and displays the content shared by at least two HMDs on the screen of the display unit 512 Control is performed through user-specific interaction in the viewing area.

In addition, the controller 518 may receive a user-specific location information based on the user's GPS information acquired through the GPS in the sensor unit 520 provided in the HMD, through the activation of items for which annotation is supported from the predetermined menu of the menu processing unit 516. [ An annotation or pre-stored annotation provided through the communication unit 524 is applied to the 2D map viewing area to support the augmented reality service.

Accordingly, the 2D map viewing area is transformed into a viewing area in which the augmented reality is supported, and the HMD viewing area in which the annotation related item is activated, from the HMD of at least one other user paired with the annotated information in the augmented reality viewing area As shown in FIG.

At this time, the control unit 518 may be configured such that a first service related to the item formed in the divided area is provided to the first location in the viewing area, and a second service related to the item formed in the divided area is provided to the second location, And controls to provide a second service different from the first service.

Here, the first and second services may be configured such that the screen data of the HMD of another user paired with the HMD of the first user or paired according to the establishment of the communication connection under the control of the controller 518 is mirrored to the HMD of the first user .

Under the control of the controller 518, the menu processing unit 516 processes a predetermined menu such that the service-specific items supported by the HMD are listed and overlaid on the screen on which the content selected by the user is displayed.

Although the exemplary embodiments of the present invention have been disclosed for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. Without departing from the scope of the present invention. Accordingly, the scope of the present invention should not be limited by the illustrated embodiments, but should be determined by equivalents of the claims and the claims.

510: camera 512: display unit
514: Search unit 516: Menu processing unit
518: Control section 520:
522: content unit 524: communication unit

Claims (13)

A method of providing an information service of a terminal equipped with an HMD (Head Mounted Display)
Displaying a predetermined menu corresponding to an event detected through a user interrupt in an HMD viewing area of a first user,
Broadcasting an item for supporting a screen sharing service among at least one or more other users from the displayed menu and displaying contents to be displayed in a pairing HMD of another user;
And controlling, through interaction with each user of the HMD, a viewing area of a screen broadcasted from the HMD of the first user and shared by at least one or more HMDs, Information service method. The method according to claim 1,
Wherein the service-specific items supported by the HMD are listed and overlaid on a screen on which the content selected by the user is displayed, and displayed. The apparatus of claim 1,
A plurality of divided areas are formed, corresponding items are formed in each of the divided areas,
Wherein a first location in the viewing area is provided with a first service associated with the item formed in the divided area and a second location is associated with the item formed in the divided area and a second service different from the first service is provided The method of claim 1, further comprising the step of providing the information providing service of the terminal with the HMD. The method of claim 3,
The item is content displayed in the viewing area of the HMD of the first user or another user,
The first service is a service in which an actual three-dimensional space input through a camera provided in the HMD of the first user is displayed,
Wherein the second service includes a service for displaying an actual three-dimensional space input through a camera provided in an HMD of another user that is previously paired with the HMD of the first user. Information service method. 5. The method of claim 4, wherein the first and second services include:
Wherein the screen data of the HMD of another user paired with the HMD of the first user or established according to the establishment of the communication connection is mirrored by the HMD of the first user. The method as claimed in claim 5, wherein the occurrence of a preset event is detected during the mirroring, and the mirroring restriction is determined according to the detected event type. 4. The method as claimed in claim 3, wherein the items formed at the first and second positions are capable of being moved up, down, left, and right through user interrupts. The method of claim 1,
An image input through a camera, a map image, an actual three-dimensional spatial image based on a line-of-sight path of an HMD user, and augmented reality information superimposed on each of the images, Provided services. 2. The method of claim 1, wherein the location-based 2D map viewing region for each user obtained through the GPS provided in the HMD is transformed into an augmented reality viewing region through activation of an item for which an annotation is supported from the preset menu, Collecting information annotated in the augmented reality viewing area from the HMD of at least one other paired user and integrating and displaying the annotation related item in the activated HMD viewing area. The information providing service method of the terminal. An information providing service apparatus of a terminal equipped with an HMD (Head Mounted Display)
A sensor unit and a photographing unit,
A touch screen having a plurality of divided areas, each divided area displaying a viewing area in which a corresponding item is formed,
When the item for supporting the screen sharing service among at least one or more other users is activated from the displayed menu, the other user who is paired A control unit broadcasting the content to be displayed in the HMD of the HMD and controlling the content to be broadcasted through at least two users in the viewing area of the screen displayed on the screen,
And a menu processing unit for processing a predetermined menu such that the items corresponding to the services supported by the HMD are listed under the control of the control unit and the content selected from the user is overlaid on the displayed screen. Information providing service apparatus of the terminal. 11. The apparatus according to claim 10,
Based 2D map viewing region obtained by the GPS provided in the HMD through the activation of an item for which an annotation is supported from the predetermined menu of the menu processing unit is transformed into an augmented reality viewing region, Wherein the annotation-related item is integrated and displayed in the HMD viewing area in which the annotation related item is activated, from the HMD of at least one other user paired with the information annotated in the viewing area. 11. The apparatus according to claim 10,
Wherein a first location in the viewing area is provided with a first service associated with the item formed in the divided area and a second location is associated with the item formed in the divided area and a second service different from the first service is provided Wherein the control information is provided to the terminal. 13. The method of claim 12, wherein the first and second services include:
Wherein the screen data of the HMD of another user paired with the HMD of the first user or established according to the communication connection establishment is mirrored by the HMD of the first user under the control of the controller, Provided service device.

Images