KR102461024B1 - Head mounted display and method for executing action in virtual environment using the same - Google Patents

Abstract

A head mounted display according to an embodiment includes a sensor unit detecting a movement of a wearer, an image display unit displaying a virtual space image and motion information indicating the detected motion, a plurality of motion information and the plurality of motion information. A storage unit for storing information on a motion mapped to each of the motion information of the dog, a manipulation unit for receiving a confirmation input from the wearer, and information on the motion mapped to the displayed motion information are obtained from the storage unit, and a control unit for controlling the operation according to the obtained information to be performed or not performed on the virtual space image according to the confirmation input received by the manipulation unit.

Description

HEAD MOUNTED DISPLAY AND METHOD FOR EXECUTING ACTION IN VIRTUAL ENVIRONMENT USING THE SAME

The present invention relates to a head mounted display and a method for performing an operation in a virtual space using the same.

As online community activities become more common, virtual space has become a space for realizing human relationships and desires beyond the purpose of information search. Users can use various services such as information retrieval, e-commerce or communication in the virtual space.

An avatar is a medium representing a user in this virtual space. A user can continuously communicate with another's avatar by using his/her avatar in a virtual space. Such avatars are usefully used in various fields such as teleconference.

On the other hand, in communication in the virtual space, the visual element occupies a relatively large proportion. Since an avatar is a visual representation of a user in a virtual space, various studies are being conducted on a method of visually modeling an avatar for the purpose of effective communication or communication. For example, studies have been conducted to implement an avatar's mouth shape to match the voice of an avatar in real time, based on a user's voice.

Korean Patent Laid-Open Publication No. 10-2011-0002005 (published on Jan. 6, 2011)

An object of the present invention is to allow a user to reflect his/her intention through an avatar in a virtual space. For example, a technique for allowing a user to express their mood or expression through an avatar or a technique for allowing a user to perform a predefined operation on a specific object using the avatar is provided.

However, the problems to be solved of the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

A head mounted display according to an embodiment includes a sensor unit detecting a movement of a wearer, an image display unit displaying a virtual space image and motion information indicating the detected motion, a plurality of motion information and the plurality of motion information. A storage unit for storing information on a motion mapped to each of the motion information of the dog, a manipulation unit for receiving a confirmation input from the wearer, and information on the motion mapped to the displayed motion information are obtained from the storage unit, and a control unit for controlling the operation according to the obtained information to be performed or not performed on the virtual space image according to the confirmation input received by the manipulation unit.

A method for performing an operation in a virtual space image according to an embodiment is performed by a head mounted display, displaying a virtual space image, detecting a wearer's movement, and the detected movement. Displaying motion information representing a motion, and applying information on the motion mapped to the displayed motion information to a plurality of motion information pre-stored in the head mounted display and an operation mapped to each of the plurality of motion information The method includes: obtaining from among information on , accepting a confirmation input from the wearer; and controlling an operation according to the obtained information according to the confirmation input to be performed or not performed in the virtual space image.

According to an embodiment, the wearer may express his/her mood or expression through an avatar in a virtual space. In addition, the wearer may cause a predefined operation to be performed on a specific object existing in the virtual space image.

1 illustrates wearers wearing the head mounted display.
FIG. 2 shows an exemplary form of a virtual space or virtual reality experienced by the wearers shown in FIG. 1 .
3 exemplarily illustrates a configuration of a head mounted display according to an embodiment.
FIG. 4 illustrates, for example, a movement of the wearer that can be detected by the sensor unit shown in FIG. 3 .
FIG. 5 exemplarily illustrates a screen provided by the image display unit shown in FIG. 3 to a wearer wearing the head mounted display shown in FIG. 3 .
FIG. 6 shows an example of information stored by the storage unit shown in FIG. 3 .
7 exemplarily illustrates an avatar representing a wearer's facial expression in a virtual space, according to an embodiment.
FIG. 8 exemplarily shows the external appearance of the manipulation unit shown in FIG. 3 .
9 exemplarily illustrates a procedure of a method for performing an operation in a virtual space according to an embodiment.
10 exemplarily illustrates a process in which a specific object is selected in a virtual space by a wearer's movement.
FIG. 11 shows another example of information stored by the storage unit shown in FIG. 3 .

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification.

1 illustrates that the wearers 10 wear a head mounted display (HMD) 100 . Each of the wearers 10 may exist in a real space adjacent to each other or in the same real space or spaced apart from each other.

FIG. 2 shows an exemplary form of a virtual circumstance or virtual reality (VR) that the wearers 10 shown in FIG. 1 are experiencing. Referring to FIG. 2 , each wearer 10 is represented as an avatar in a virtual space. At this time, regardless of whether the wearers 10 shown in FIG. 1 are adjacent to each other, exist in the same real space, or exist in a real space spaced apart from each other, the avatars of the wearers 10 may exist in the same virtual space. have.

The wearers 10 may conduct a meeting through their avatars in a virtual space or buy and sell things. In addition, the wearers 10 may express their own information or opinions through the avatar, and communicate with the avatars of other wearers 10 . As shown in FIG. 2 , the wearer 10's own name or a name assigned to the avatar may be displayed on the avatar.

The wearers 10 may allow a predefined motion to be performed through an avatar in a virtual space. The predefined operation may include, for example, making one's avatar have various facial expressions in the virtual space, and performing a predefined operation on a specific object disposed in the virtual space. To this end, the wearers 10 may use the head mounted display 100 they are wearing. Hereinafter, the head mounted display 100 will be described in detail.

FIG. 3 exemplarily shows the configuration of the head mounted display 100 shown in FIG. 1 . Referring to FIG. 3 together with FIGS. 1 and 2 , the head mounted display 100 includes a sensor unit 110 , an image display unit 120 , a storage unit 130 , a manipulation unit 140 , and a control unit 150 . However, the configuration of the head mounted display 100 is not limited to that illustrated in FIG. 3 . For example, although not shown in FIG. 3, the head mounted display 100 may include a communication unit for communicating with a smart device such as a smartphone, a power supply unit, and a wearing unit providing means to be worn on the head of the wearer 10. can According to an embodiment, the head mounted display 100 is combined with a smart device such as a smart phone and has the above-described configuration, for example, the sensor unit 110 , the image display unit 120 , the storage unit 130 , the operation unit 140 or the control unit. (150) and the like may be implemented.

Among them, the sensor unit 110 is mounted on the head mounted display 100 to detect the movement of the head mounted display 100 itself. Here, the head mounted display 100 is mounted on the head of the wearer 10 . Accordingly, the movement detected by the sensor unit 110 may be a head movement of the wearer 10 . FIG. 4 illustrates, for example, a head movement of the wearer 10 that the sensor unit 110 can detect, but is not limited to the movement illustrated in FIG. 4 . The sensor unit 110 may be implemented as an inertial measurement unit (IMU), but is not limited thereto.

Referring back to FIG. 3 , the image display unit 120 displays an image of the virtual space to be provided to the wearer 10 . 5 exemplarily illustrates a virtual space image provided by the image display unit 120 to the wearer 10 .

Referring to FIG. 5 , various objects A and B existing in the virtual space may be displayed on the image display unit 120 .

In addition, the wearer 10's own movement detected by the sensor unit 110, for example, the head movement of the wearer 10 may be displayed on the image display unit 120 . At this time, as shown in the lower part of FIG. 5 , the head movement may be displayed in the shape of a gyro 121 . For example, when the wearer 10 nods the head, the gyro 121 may also move in the direction in which the wearer 10 nods the head.

In the center of the screen of the image display unit 120 , a center mark 123 indicating the center of the screen and also indicating the gaze direction of the wearer 10 may be disposed. The center mark 123 may be used later when the wearer 10 selects a specific object, which will be described later with reference to FIG. 10 .

A predefined mark 122 may be displayed on the image display unit 120 . The situation in which the mark 122 is displayed means that the controller 150, which will be described later, is ready to accept and interpret the wearer's 10 head movement as a command or a specific intention of the wearer 10 . That is, when the mark 122 is being displayed on the image display unit 120 , the head movement of the wearer 10 sensed by the sensor unit 110 is a command of the wearer 10 or a specific intention by the control unit 150 . On the other hand, when the mark 122 is not displayed on the image display unit 120 , the head movement of the wearer 10 may not be interpreted as a command or a specific intention of the wearer 10 . The indicia 122 may be a specific image or text. Such a mark 122 may be displayed only when the wearer 10 manipulates the manipulation unit 140 to be described later, which will be described later.

On the other hand, according to an embodiment, the image display unit 120 that does not introduce the above-described mark 122 itself may be implemented. (150) will be described in more detail while explaining.

The storage unit 130 stores data and may be implemented as a memory or the like. 6 illustrates data stored in the storage unit 140 as an example. Referring to FIG. 6 , a plurality of predefined motion information and motion information mapped to each of the plurality of motion information may be stored in the storage unit 140 . That is, the storage unit 130 may store information on which action should be performed when a specific movement is detected by the sensor unit 110 .

Here, the motion information represents the motion of the wearer 10 . For example, the movement information may include head movements of the wearer 10 for each type as shown in FIG. 4 . The motion information indicates motions that can be performed in a virtual space image. For example, in the motion information, the avatar 124 of the wearer 10 makes a specific facial expression (a happy expression, a sad expression, a surprised expression, etc.), and the avatar 124 of the wearer 10 as shown in FIG. Displaying the emoticon 125 representing the back, allowing the wearer 10's avatar to display the wearer's 10 mood in text, the movement information currently stored in the storage unit 130 and each movement information Displaying the motion information mapped to the wearer 10 in the form of a list may be included.

The manipulation unit 140 is configured to receive an input from the wearer 10 . FIG. 8 shows an external shape of the manipulation unit 140 by way of example, and the manipulation unit 140 having such an external shape may be connected to the head mounted display 100 in a wired or wireless manner. However, unlike shown in FIG. 8 , the manipulation unit 140 may be implemented in the form of a button on the head mounted display 100 . Meanwhile, in the following description, the operation unit 140 will be described on the assumption that it has the external shape shown in FIG. 8 .

Referring to FIG. 8 , the manipulation unit 140 may include an input button 141 . When the wearer 10 presses the input button 141 or when the pressed state is released, a signal may be generated respectively, and each signal may be transmitted to the control unit 150 to be described later. The control unit 150 may receive a signal from the operation unit 140 to perform a predefined function, which will be described in more detail below.

Referring to FIG. 3 , the head mounted display 100 includes a controller 150 . The controller 150 may be implemented by a memory for storing instructions programmed to perform a function to be described below and a microprocessor for executing these instructions.

When the input button 141 of the manipulation unit 140 is pressed, the manipulation unit 140 may transmit a first signal to the control unit 150, and in this case, the control unit 150 transmits the above-mentioned information to the image display unit 120 based on the first signal. One mark 122 can be controlled to be displayed. This mark 122 is ready to interpret the movement of the wearer 10 sensed by the sensor unit 110 after the mark 122 is displayed by the control unit 150 as a command or a specific intention of the wearer 10 could mean that Hereinafter, the manipulation of the wearer 10 applied to the manipulation unit 140 so that the image display unit 120 displays the mark 122 will be referred to as 'ready input'.

In addition, when the input button 141 of the manipulation unit 140 is pressed and released, the manipulation unit 140 may transmit a second signal to the control unit 150 , and in this case, the control unit 150 may control the second signal Based on , motion information mapped to the motion of the wearer 10 sensed by the sensor unit 110 is obtained from the storage unit 130 , and then an operation corresponding to the motion information may be controlled to be performed. That is, the operation itself of pressing and releasing the input button 141 of the manipulation unit 140 may mean confirming that a predefined operation mapped to the movement of the wearer 10 is performed. The manipulation of the wearer 10 applied to the manipulation unit 140 so that the above-described predefined operation is performed will be referred to as a 'confirmation input'. Here, when the input button 141 of the manipulation unit 140 is pressed and not released for a certain period of time, for example, 5 seconds, the manipulation unit 140 may transmit a third signal to the control unit 150, in this case Based on the third signal, the control unit 150 may control not to perform any operation stored in the storage unit 130 separately from sensing the movement of the wearer 10 .

Meanwhile, according to an embodiment, the operation of the image display unit 120 to display the mark 122 may not be performed. That is, in this case, a 'ready input' is not required, and thus the wearer 10 does not operate the manipulation unit 140 for the ready input. In this embodiment, regardless of the mark 122, the sensor unit 110 detects the movement of the wearer 10, the image display unit 120 displays motion information on the detected movement, and the control unit 150 may interpret this motion information as a command or a specific intention of the wearer 10 . If the motion information is inaccurate or is motion information not stored in the storage unit 130 , no operation may be performed.

Hereinafter, a process in which a predefined operation is performed in a virtual space image by the head mounted display 100 will be described with reference to FIG. 9 .

Referring to FIG. 9 , first, the image display unit 120 may display a virtual space image to be provided to the wearer 10 ( S100 ).

After that, when the input button 141 of the manipulation unit 140 is pressed, the manipulation unit 140 accepts it as a ready input (S110), and the image display unit 120 displays a predefined mark 122 in the virtual space image. It can be done (S120). However, if the input button 141 of the manipulation unit 140 is not pressed, the image display unit 120 does not display the predefined mark 122 in the virtual space image. Here, depending on the embodiment, steps S110 and S120 may not be performed, and in this case, the following steps S130 may be performed directly from step S100.

The motion of the wearer 10 detected by the sensor unit 110 (S130) may be displayed as motion information on the image display unit 120 (S140). The motion information may be displayed in a gyro shape.

When the wearer 10 releases the input button 141 of the manipulation unit 140 within a predefined time, the manipulation unit 140 accepts it as a confirmation input (S150). In this case, the controller 150 may obtain the motion information mapped to the motion information displayed on the image display unit 120 from the storage unit 130 and then perform an operation according to the motion information (S170). The operation here is that the avatar of the wearer 10 makes a specific facial expression (a happy expression, a sad expression, a surprised expression, etc.), and as shown in FIG. 125), allowing the wearer 10's avatar to display the wearer's 10 mood as text, and motion information currently stored in the storage unit 130 and an operation mapped to each motion information Displaying information to the wearer 10 in the form of a list may be included.

On the other hand, when the wearer 10 does not release the input button 141 of the manipulation unit 140 even after a predefined time elapses or releases it after a predefined time, the control unit 150 receives it as unconfirmed take in In this case, the control unit 150 may control the image display unit 120 to display a message requesting the movement again to the wearer 10 ( S160 ).

Here, if steps S110 and S120 are not performed according to an embodiment, the confirmation input of step S150 may be input by the wearer 10 pressing the input button 141 of the manipulation unit 140 .

As described above, according to an embodiment, the wearer may express his or her mood or expression through an avatar in a virtual space. In this case, it is possible to accurately express one's mood or expression through the avatar by the image display unit indicating the movement of one's own head and the operation unit used to confirm whether the movement of the head is intended by the user.

Meanwhile, in the above-described embodiment, the mood or expression of the wearer 10 is displayed through the avatar. Alternatively or in addition to this, a predefined operation may be performed on a specific object existing in the virtual space image. This will be described with reference to FIGS. 10 and 11 .

10 exemplarily illustrates a process in which any one object is selected from among a plurality of objects existing in a virtual space image. Referring to FIG. 10 , a center mark 123 may be displayed on the image display unit 120 . The center mark 123 represents the center of the screen in the center of the screen of the image display unit 120 and also represents the gaze direction of the wearer 10 .

The wearer 10 may make the center mark 123 move in the virtual space image by using his/her head movement. For example, when the wearer 10 rotates his/her head left and right, the trajectory 125 may be as shown in FIG. 10 . In this case, the controller 150 may recognize the object A on the path through which the corresponding trajectory 125 passes as the object selected by the wearer 10 . That is, a plurality of objects exist in the virtual space image, and there may also be at least one object displayed on the image display unit 120 . In an embodiment, as a method of selecting any one of the objects, the wearer 10 owns A method using the trajectory 125 of the head movement is presented. However, the shape of the trajectory 125 shown in FIG. 10 is exemplary, and trajectories of various shapes may be defined in advance.

In this embodiment, the storage unit 130 may store motion information for all objects existing in the virtual space image, that is, each of a plurality of objects, and motion information mapped to each motion information. 11 shows an example of the form of such information in the form of a list. Referring to FIG. 11 , a plurality of different motion information is set and stored. In addition, an operation performed by a specific movement for each object may be different or the same. For example, when the movement is a, an operation of opening the corresponding object A may be performed on the object A, but the operation of drinking the corresponding object B may be performed on the object B. The wearer 10 should know in advance which head movements are necessary to perform which movements.

Hereinafter, a detailed process in which this embodiment is implemented will be described as an example with reference to FIG. 9 again.

First, the image display unit 120 may display a virtual space image to be provided to the wearer 10 (S100).

After that, when the input button 141 of the manipulation unit 140 is pressed, the manipulation unit 140 accepts it as a ready input (S110), and the image display unit 120 displays a predefined mark 122 in the virtual space image. It can be done (S120). However, if the input button 141 of the manipulation unit 140 is not pressed, the image display unit 120 does not display the predefined mark 122 in the virtual space image. Here, depending on the embodiment, steps S110 and S120 may not be performed, and in this case, the following steps S130 may be performed directly from step S100.

The motion of the wearer 10 detected by the sensor unit 110 (S130) may be displayed as motion information on the image display unit 120 (S140). The motion information may be displayed in a gyro shape. In addition, this movement may be utilized as information for selecting any one of a plurality of objects existing in the virtual space image by the controller 150 . That is, when the trajectory of the center mark 123 is changed by movement, an object on the trajectory may be selected.

When the wearer 10 releases the input button 141 of the operation unit 140 within a predefined time, and thus the confirmation input is accepted by the operation unit 140 (S150), the control unit 150 displays the displayed image on the image display unit 120 (S150). After the motion information mapped to the motion information is obtained from the storage unit 130, an operation according to the motion information may be performed (S170). Here, the operation may be an operation stored in advance in the selected specific object, for example, an operation to open, listen to, throw, drink, or discard the object, and also the movement information and each movement information currently stored in the storage unit 130 . Displaying the motion information mapped to the wearer 10 in the form of a list may be included.

On the other hand, if the wearer 10 does not release the input button 141 of the manipulation unit 140 even after a predefined time elapses or releases it after a predefined time, the control unit 150 may not accept it as not confirmed. In this case, the control unit 150 may control the image display unit 120 to display a message requesting the movement again to the wearer 10 (S160).

Here, if steps S110 and S120 are not performed according to the embodiment, the confirmation input of step S150 may be accepted by the operation unit 140 by the wearer 10 pressing the input button 141 of the operation unit 140 . have.

As described above, according to an embodiment, the wearer may use an avatar to perform a predefined operation on a specific object existing in a virtual space image.

The above description is merely illustrative of the technical idea of the present invention, and various modifications and variations will be possible without departing from the essential quality of the present invention by those skilled in the art to which the present invention pertains. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

According to an embodiment, the wearer of the head mounted display may reflect his/her intention through the avatar in the virtual space.

100: head mounted display

Claims (12)

In the head mounted display,
A sensor unit that detects the movement of the wearer,
A virtual space image and an image display unit for displaying motion information representing the sensed motion, wherein an avatar corresponding to the wearer is predefined;
The head mounted display,
a storage unit for storing a plurality of motion information and information on the motion of the avatar mapped to each of the plurality of motion information;
a manipulation unit for receiving a confirmation input from the wearer;
information on the motion mapped to the displayed motion information is obtained from the storage unit, and according to the confirmation input received by the manipulation unit, the motion of the avatar according to the obtained information is performed or not performed in the virtual space image including a control unit to control,
The motion of the avatar mapped to each of the plurality of motion information includes:
In the virtual space image, including displaying the expression of the avatar corresponding to the wearer,
Head mounted display. The method of claim 1,
The video display unit,
Displaying movement information corresponding to the wearer's head movement among the plurality of movement information
Head mounted display. 3. The method of claim 2,
The video display unit,
Displaying movement information corresponding to the wearer's head movement in a gyro shape
Head mounted display. delete The method of claim 1,
The operation stored in the storage unit,
Displaying the plurality of motion information and information on motion mapped to each of the plurality of motion information to be provided to the wearer in the virtual space image
Head mounted display. The method of claim 1,
The manipulation unit receives a ready input from the wearer,
The image display unit displays a predefined mark on the virtual space image, and then displays movement information on the movement sensed by the sensor unit, performed by the wearer.
Head mounted display. A method for performing an operation in a virtual space image, the method being performed by a head mounted display,
displaying the virtual space image;
detecting the wearer's movement; and
displaying motion information indicating the detected motion, wherein an avatar corresponding to the wearer is predefined;
A method for performing an operation in the virtual space image,
obtaining information on the motion of the avatar mapped to the displayed motion information from a plurality of motion information pre-stored in the head mounted display and a motion mapped to each of the plurality of motion information;
receiving a confirmation input from the wearer;
and controlling the operation of the avatar according to the obtained information to be performed or not performed in the virtual space image according to the confirmation input;
The motion of the avatar mapped to each of the plurality of motion information includes:
In the virtual space image, including displaying the expression of the avatar corresponding to the wearer,
A method for allowing an action to be performed in a virtual space image. 8. The method of claim 7,
The step of displaying the motion information comprises:
Displaying movement information corresponding to the wearer's head movement among the plurality of movement information
A method for allowing an action to be performed in a virtual space image. 9. The method of claim 8,
The step of displaying the motion information comprises:
Displaying movement information corresponding to the wearer's head movement in a gyro shape
A method for allowing an action to be performed in a virtual space image. delete 8. The method of claim 7,
The operation mapped to each of the plurality of pre-stored motion information includes:
Displaying the plurality of motion information and information on motion mapped to each of the plurality of motion information to be provided to the wearer through the virtual space image
A method for allowing an action to be performed in a virtual space image. 8. The method of claim 7,
The method is
accepting a ready input from the wearer;
Further comprising the step of displaying a predefined mark on the virtual space image,
The step of displaying the motion information comprises:
Displaying motion information about a motion detected by the wearer after the mark is displayed on the virtual space image
A method for allowing an action to be performed in a virtual space image.

Images