KR102664782B1 - Electronic system and electronic device for controlling virtual reality space and operating method thereof - Google Patents

Abstract

An electronic system, an electronic device, and a method of operating the same for controlling a virtual reality space are disclosed. The disclosed electronic system includes an electronic device and a user that receive a virtual reality space and/or control of an object within the virtual reality space through one or more touch gestures, and display a first screen according to a corresponding first view frustum in the virtual reality space. It is worn on the user's head and includes an HMD that displays a second screen in the virtual reality space according to the direction of the user's head at a viewpoint corresponding to the second view frustum in the virtual reality space. The virtual display of the electronic device displayed on the second screen according to the direction of the head is synchronized with the actual display of the electronic device. When the user performs an action of inputting one or more touch gestures to the virtual display displayed on the second screen, the one or more touch gestures are input to the actual display of the electronic device to control the virtual reality space and/or object.

Description

Electronic system, electronic device, and method of operation thereof for controlling virtual reality space {ELECTRONIC SYSTEM AND ELECTRONIC DEVICE FOR CONTROLLING VIRTUAL REALITY SPACE AND OPERATING METHOD THEREOF}

The description below relates to an electronic system, an electronic device, and a method of operating the same that perform control over a virtual reality space.

Virtual Reality (VR) refers to a virtual world rather than a reality created using computers, etc. As virtual reality technology has recently developed, users can experience various experiences in virtual reality space. Additionally, users can interact with other users in virtual reality.

The electronic system according to one embodiment receives a virtual reality space and/or control of an object within the virtual reality space through one or more touch gestures, and displays a corresponding first view frustum within the virtual reality space. An electronic device that displays a first screen according to a frustum and is worn on the user's head, and a second screen in the virtual reality space according to the direction of the user's head at a viewpoint corresponding to the second view frustum in the virtual reality space and a head mounted display (HMD) that displays, wherein the virtual display of the electronic device displayed on the second screen according to the direction of the head is synchronized with the actual display of the electronic device, and the user displays a virtual display on the second screen. When an operation of inputting one or more touch gestures to the displayed virtual display is performed, the one or more touch gestures are input to the actual display of the electronic device to control the virtual reality space and/or the object.

The virtual display of the electronic device displayed on the second screen may display the first screen displayed on the actual display of the electronic device.

One or a combination of two or more of the size, position, and direction of the virtual display of the electronic device displayed on the second screen may be synchronized with the actual display of the electronic device.

The user's tracked hand is displayed on the second screen along with the virtual display of the electronic device, and the user's action of inputting a touch gesture on the virtual display may result in the touch gesture being input on the actual display.

The electronic device performs movement and/or rotation with respect to the first view frustum in the virtual reality space based on one or more touch gestures input from the user, and movement and/or rotation with respect to the first view frustum. Rotation may be performed within preset boundaries within the virtual reality space.

The electronic device performs a sketch on the screen plane of the first view frustum in the virtual reality space based on the pen drawing input to the actual display, and displays the sketched screen plane as the first screen. You can.

The electronic device may perform one or a combination of two or more of movement, rotation, and size adjustment of the object on the screen plane based on one or more touch gestures for the object created by the sketch.

The electronic device may determine a projection surface of the object created by the sketch based on the gaze direction of the first view frustum when performing the sketch.

When a creation command for the object is input from the user, the electronic device may create the object on the projection surface and place it in the virtual reality space.

Based on one or more touch gestures for a card containing an object in the virtual reality space, the electronic device performs one or a combination of two or more of movement, rotation, and size adjustment for the object on a first reference plane containing the card. It can be done.

The electronic device performs movement and/or rotation of the object on a second reference plane containing the pedestal based on one or more touch gestures with respect to the pedestal of the card containing the object in the virtual reality space, and the pedestal The card may be generated parallel to the floor at a position where the card is projected onto the floor of the virtual reality space.

The electronic device may control the height of the card in the virtual reality space based on a second touch gesture for the card while a first touch gesture for the base of the card including an object in the virtual reality space is input. there is.

The electronic device performs a rotation on the object based on a second touch gesture on an area other than the card or the pedestal while a first touch gesture is input on the pedestal of the card containing the object in the virtual reality space. can do.

One or a combination of two or more of movement, rotation, and size adjustment of the object according to one or more touch gestures input from the user may be performed within a preset boundary within the virtual reality space.

The first screen and/or the second screen displays a third view frustum corresponding to a second electronic device connected to the virtual reality space at a position corresponding to the second electronic device at a viewpoint of the third view frustum. Direction can be indicated.

The first screen and/or the second screen responds when a sketch is performed on the second screen plane of the third view frustum based on a pen drawing input to the actual display of the second electronic device, The sketched second screen plane may be displayed.

The electronic device may perform one or a combination of two or more of movement, rotation, and size adjustment for the object based on one or more pinch gestures for the object placed in the virtual reality space input from the user wearing the HMD. You can.

A method of operating an electronic device according to an embodiment includes displaying a first screen according to a first view frustum corresponding to the electronic device in a virtual reality space and displaying a first screen according to the first view frustum corresponding to the electronic device in the virtual reality space and/or an object in the virtual reality space. In response to a case where the user controlling the electronic device wears the HMD on the head, the HMD displays a second view frustum corresponding to the second view frustum in the virtual reality space. A second screen in virtual reality space is displayed according to the direction of the user's head at a viewpoint, and the virtual display of the electronic device displayed on the second screen according to the direction of the user's head is synchronized with the actual display of the electronic device. , when the user performs an operation of inputting one or more touch gestures to the virtual display displayed on the second screen, the one or more touch gestures are input to the actual display of the electronic device to enter the virtual reality space and / or the object. control is carried out.

According to one embodiment, a user can easily create and control objects in a virtual reality space through touch gestures and pen drawings. Additionally, the user can easily control the corresponding view frustum in the virtual reality space through one or more touch gestures.

According to one embodiment, the physical display of the electronic device and the virtual display of the electronic device displayed on the HMD are synchronized with each other, the virtual display displayed on the HMD displays a screen displayed on the physical display of the electronic device, and the virtual display of the electronic device on the HMD is displayed. By displaying the display and the tracked user's hand, the user can input a touch gesture on the actual display of the electronic device without any discomfort through the user's action of inputting a touch gesture on the virtual display.

FIG. 1 is a diagram for explaining an electronic device according to an embodiment.
FIG. 2 is a diagram for explaining a virtual reality space and objects within the virtual reality space according to an embodiment.
Figures 3 to 5 are examples for explaining camera manipulation-related operations according to an embodiment.
6 to 11 are examples for explaining operations related to object creation according to an embodiment.
12 to 17 are examples for explaining operations related to objects created perpendicular to the floor of a virtual reality space according to an embodiment.
18 to 23 are examples for explaining operations related to objects created horizontally on the floor of a virtual reality space according to an embodiment.
Figures 24 and 25 are examples for explaining operations related to manipulating a card including an object according to an embodiment.
Figure 26 is an example to explain a multi-user related operation according to an embodiment.
27 to 29 are examples for explaining HMD-related operations according to an embodiment.
Figure 30 is a diagram showing a method of operating an electronic device according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, but are not intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 is a diagram for explaining an electronic device according to an embodiment.

According to one embodiment, the electronic device 110 may allow an object to be sketched in a three-dimensional virtual reality space based on the user's body movement (eg, touch gesture and/or hand movement, etc.) and/or pen drawing. For example, the electronic device 110 may be implemented in the form of a tablet, but is not limited to the above-described example. Through the electronic device 110, a user can place a sketched object in a three-dimensional virtual reality space, control the position, size, rotation, etc. of the object within the virtual reality space, and control the object placed within the virtual reality space. By controlling the viewing angle of the camera, you can view objects from various perspectives. Through these controls, users can sketch, place, and move various objects within the three-dimensional virtual reality space, and observe the objects from various angles, allowing for a variety of experiences distinct from the real world space.

Electronic device 110 may include a display and/or sensor. The display can receive the user's touch gestures and pen input. The display may be a touch screen that can receive input from a user's touch gestures or pen drawings. The sensor detects the user's hand movements, etc., and, as will be described later, can also detect the position and/or direction of the HMD when the user wears the HMD.

FIG. 2 is a diagram for explaining a virtual reality space and objects within the virtual reality space according to an embodiment.

Referring to FIG. 2, objects arranged in a virtual reality space are exemplarily shown. In virtual reality space, objects may be drawn on the surface represented by the card 210. Objects are drawn by the user and can have various forms. The card 210 represents an area containing an object and is illustratively shown as a rectangle in FIG. 2, but is not limited to the above-described example and may have various shapes. The card 210 may be perpendicular to the floor of the virtual reality space shown in a grid form at the bottom of FIG. 2, but the embodiment is not limited thereto, and may be horizontal to the floor as will be described later. The stand 220 of the card 210 may be a circle centered on the center of the card 210 when it is projected on the floor of the virtual reality space, but is not limited to the above-described example. The pedestal 220 may be placed on the floor of the virtual reality space. As will be explained later, different control operations may be performed depending on whether the user's touch gesture is input to the object 210, the pedestal 220, or an area other than the object 210 or the pedestal 220. there is. This will be explained in detail through the drawings.

Figures 3 to 5 are examples for explaining camera manipulation-related operations according to an embodiment.

The display of the electronic device 110 of FIG. 1 may display a screen according to the corresponding view frustum in the virtual reality space. The view frustum is a virtual structure corresponding to the camera viewpoint for determining the screen to be provided to the user in the virtual reality space, which is explained in detail with reference to FIG. 26. The user can control the screen displayed on the display of the electronic device by controlling the position and/or angle of the view frustum through one or more touch gestures.

Referring to Figure 3, control of the view frustum based on one touch gesture is exemplarily shown. In step 310, the user may touch the display of the electronic device with one finger. In step 320, the user may touch the display and move the finger to another location on the display. In summary, the user can input a drag gesture based on one finger, and in response, an action of rotating the gaze direction from a fixed camera viewpoint can be performed. For example, the position of the view frustum in the virtual reality space does not change, and the gaze direction of the view frustum may rotate depending on the drag direction.

Referring to Figure 4, control of the view frustum based on two touch gestures is exemplarily shown. In step 410, the user may touch the display of the electronic device with two fingers.

In step 420, the user can move two fingers in parallel to different positions on the display while touching the display. In other words, the user can input a drag gesture in which the distance between the two fingers is maintained at a certain level, and in response, the camera viewpoint can be moved parallel to the drag direction. For example, the position of the view frustum in the virtual reality space may move parallel to the drag direction.

Additionally, in step 430, the user may touch the display and move the two fingers to different positions on the display in a direction in which the two fingers move away from each other. In other words, the user can input a drag gesture that increases the distance between the two fingers, and in response, the camera viewpoint can be moved forward. This operation can increase the size of the object displayed on the display. For example, the position of the view frustum in the virtual reality space may move forward in the direction of gaze. Conversely, if the user inputs a drag gesture that brings the distance between the two fingers closer, an action may be performed to move the camera viewpoint back, and this action may cause the size of the object displayed on the display to become smaller and the view plus Objects that were located behind the term and were not displayed on the display may be displayed on the display by moving the view frustum.

Additionally, in step 440, the user may move two fingers to different positions on the display so that the angle formed by the two fingers changes while touching the display. In other words, the user can input a drag gesture that changes the angle formed by the two fingers, and in response, the camera viewpoint can be rotated. For example, the view frustum within the virtual reality space may rotate. For example, rotation may be performed based on a point in virtual reality space, but is not limited to the above-described example.

The touch gestures described above may be input independently or in combination, and correspondingly, movement and rotation of the view frustum may be performed independently or in combination.

Referring to FIG. 5, an example is shown to explain limitations due to boundaries of virtual reality space. Depending on the embodiment, the size and shape of the virtual reality space may be determined in advance, and boundaries may be set according to the size and shape. For example, the size and shape of the virtual reality space can be set by the user or the system. Movement and/or rotation of the view frustum in virtual space may be performed within preset boundaries in virtual reality space based on one or more touch gestures described above. For example, if the user touches the display of the electronic device with two fingers in step 510 and moves the two fingers in a direction beyond the boundary of the virtual reality space in step 520, the view frustum Movement is limited within the boundary, and feedback may be provided to the user that movement and/or rotation of the view frustum is limited by the boundary. Feedback can be provided in various forms, including visual, auditory, and tactile.

6 to 11 are examples for explaining operations related to object creation according to an embodiment.

Referring to FIG. 6 , examples of a touch gesture 610 and a pen input 620 for sketching an object are shown.

An interface for adjusting the pen thickness may be displayed on one side of the display of the electronic device 110 of FIG. 1. The user can control the pen thickness through the touch gesture 610. For example, in the touch gesture 610, the pen thickness can be controlled by dragging the buttons representing the pen thickness up and down.

Additionally, an interface (eg, palette) for controlling pen colors may be displayed on the other side of the display. The square cards shown on the left side of FIG. 6 may represent different colors. Users can select the color card they want to use with the pen tip.

Depending on the embodiment, the user may control the pen thickness using a touch gesture 610 based on one hand and the pen color using a pen input 620 based on the other hand.

Referring to FIG. 7, the operation of creating an object is exemplarily shown. A user can perform a sketch on the screen plane of the view frustum within the virtual reality space by inputting a pen drawing on the display of the electronic device. Steps 710 and 720 may represent an example in which a sketch is performed on the screen plane of the view frustum based on a pen drawing input to the display, and the sketched screen plane is displayed on the display. In step 730, if the direction of the pen is reversed and input is input on the display, the sketch of the input portion may be erased.

Referring to FIG. 8, control of an object based on one touch gesture is exemplarily shown. At step 810, the user can touch the object with one finger. In step 810, the user may move the finger to another location on the display while touching the object. In summary, the user can input a drag gesture based on one finger, and correspondingly, movement of the selected object can be performed on the screen plane of the view frustum.

Referring to FIG. 9, control of an object based on two touch gestures is exemplarily shown. At step 910, the user can touch the object with two fingers.

In step 920, the user may move two fingers to different positions on the display in a direction in which the two fingers move away from each other while touching the object. In other words, the user can input a drag gesture that increases the distance between the two fingers, and the size of the object can increase in response.

In step 930, the user may move two fingers to different positions on the display so that the angle formed by the two fingers changes while touching the object. In other words, the user can input a drag gesture that changes the angle formed by the two fingers, and an operation in which the object rotates can be performed in response.

Additionally, if the user moves two fingers in parallel to different positions on the display while touching the object, an operation in which the object moves in parallel in the drag direction may be performed in response.

The touch gestures described above may be input independently or in combination, and in response, one or a combination of two or more of movement, rotation, and size adjustment of the object may be performed. The movement, rotation, and scaling of objects described above can be performed on the screen plane of the view frustum.

Referring to FIG. 10, an example is shown to explain an operation of determining the direction of a projection surface on which an object created by a sketch will be projected in a virtual reality space. The electronic device may determine the direction of the projection surface of the object from the gaze direction of the view frustum. For example, the projection surface may be perpendicular to the floor of the virtual reality space, or may be horizontal. For example, if the angle between the viewing direction of the view frustum and the floor of the virtual reality space is greater than or equal to a critical angle (e.g., 45 degrees, etc.), the projection surface may be determined to be perpendicular to the floor. Conversely, if the angle between the viewing direction of the view frustum and the floor of the virtual reality space is smaller than the critical angle, the projection surface may be determined to be horizontal to the floor. However, the angle of the projection surface is not limited to the above-described example, and the projection surface may be determined at various angles with the floor depending on the embodiment. Step 1010 may represent an example of a projection surface determined to be perpendicular to the floor, and step 1020 may represent an example of a projection surface determined to be horizontal to the floor.

Referring to FIG. 11 , an example of an object being created on a projection surface based on a button displayed on the display of an electronic device is shown. When a user selects a button displayed on the display with one finger 1110, a command to create an object may be input to the electronic device. Objects can be created on the projection surface according to the creation command and placed in the virtual reality space. The angle of the projection surface can be determined to be vertical or horizontal to the floor, and the distance to the projection surface can be determined so that the sketched object touches the floor, ceiling, or boundary when projected. Since the matters described with reference to FIG. 10 can be applied to the operation of determining the projection surface, a more detailed description will be omitted. A card 1120 containing an object placed on the projection surface may be displayed.

12 to 17 are examples for explaining operations related to objects created perpendicular to the floor of a virtual reality space according to an embodiment.

Referring to FIG. 12 , control of an object based on a single touch gesture on a card 1210 is exemplarily shown. First, the user can touch the card 1210 containing an object placed in the virtual reality space with one finger. When a touch gesture is input to the card 1210 including a placed object, a first reference plane 1220 may be displayed based on the card 1210. The first reference plane 1220 may be a plane with a larger size than the card 1210 including the card 1210. For example, the first reference plane 1220 may be a large plane that extends to a preset boundary within the virtual reality space. Also, while the user touches the card 1210, he or she can move his or her finger to another position on the display. In summary, the user can input a drag gesture based on one finger, and in response, the object within the card 1210 can be moved on the first reference plane 1220.

Referring to Figure 13, control of an object based on two touch gestures for a card is exemplarily shown. In step 1310, the user may touch a card containing an object placed in the virtual reality space with two fingers. Likewise, when a touch gesture is input to a card including a placed object, a first reference plane may be displayed based on the card. In step 1320, the user can move two fingers to different positions on the display while touching the card. In summary, the user can input a drag gesture based on two fingers, and in response, the movement and/or size adjustment of the object in the card can be performed on the first reference plane.

Referring to Figure 14, control of an object based on a single touch gesture on a pedestal is exemplarily shown. The pedestal may be created parallel to the floor at a position where cards containing objects placed in the virtual reality space are projected onto the floor of the virtual reality space. At step 1410, the user can touch the base with one finger. The pedestal on which the touch gesture has been input is visually displayed differently (e.g., darker or brighter, etc.) than the pedestal on which the touch gesture has not been input, thereby providing feedback to the user that the touch gesture has been accurately input for the pedestal. In step 1420, the user may move the finger to another location on the display while touching the base. In summary, the user can input a drag gesture based on one finger, and in response, movement of an object corresponding to the pedestal can be performed on a second reference plane including the pedestal. The second reference plane is a plane of a larger size than the pedestal including the pedestal, and may be, for example, a large plane that extends to a predetermined boundary within the virtual reality space. For example, the second reference plane may correspond to the floor of the virtual reality space.

Referring to Figure 15, control of an object based on two touch gestures for a pedestal is exemplarily shown. At step 1510, the user can touch the base with two fingers. In step 1520, the user may input a drag gesture that changes the angle formed by the two fingers, and in response, rotation of the object corresponding to the pedestal may be performed on the second reference plane. In step 1530, the user may input a drag gesture in which the distance between the two fingers is maintained at a certain level, and in response, the object corresponding to the pedestal may be moved on the second reference plane.

Referring to Figure 16, control of an object based on two touch gestures for a card and a pedestal is exemplarily shown. In step 1610, the user may touch the base of the object placed in the virtual reality space with one finger and touch the card containing the object with the other finger. In step 1620, the user may input a drag gesture for the card while maintaining the touch gesture for the base, and the height of the card in the virtual reality space may be controlled correspondingly. For example, the height of the card can be controlled on a first reference plane containing the card. The height of objects included in the card can be similarly controlled.

Referring to FIG. 17, control of objects based on two touch gestures for a card and other areas is exemplarily shown. At step 1710, the user may touch the pedestal with one finger and touch a card or other area other than the pedestal with another finger. In step 1720, the user may input a drag gesture for another area while maintaining the touch gesture for the base, and rotation of the object may be performed in response.

18 to 23 are examples for explaining operations related to objects created horizontally on the floor of a virtual reality space according to an embodiment.

18, control of an object based on a single touch gesture on a card is exemplarily shown. In step 1810, the user may touch a card containing an object placed in the virtual reality space with one finger. When a touch gesture is input to a card, a first reference plane may be determined based on the card. Since the object is created horizontally on the floor, the first reference plane and the floor may be parallel. In step 1820, when the user inputs a drag gesture based on one finger, movement of the object in the card may be performed on the first reference plane in response.

19, control of an object based on two touch gestures for a card is exemplarily shown. In step 1910, the user may touch a card containing an object placed in the virtual reality space with two fingers. In step 1920, when the user inputs a drag gesture based on two fingers (e.g., a gesture of moving the two fingers apart or closer together), size adjustment of the object may be performed on the first reference plane in response. . In step 1930, when the user inputs a drag gesture based on two fingers (e.g., a gesture that changes the angle formed by two fingers), rotation of the object may be performed on the first reference plane in response. In addition, although not shown in FIG. 19, when the user inputs a drag gesture based on two fingers (e.g., a drag gesture in which the distance between the two fingers is maintained at a certain level), movement of the object in response to the drag gesture is performed on the first reference plane. It can be performed on

The drag gestures based on the two fingers described above may be input independently or in combination, and in response, one or a combination of two or more of movement, rotation, and size adjustment of the object may be performed on the first reference plane.

Referring to Figure 20, control of an object based on a single touch gesture on a pedestal is exemplarily shown. At step 2010, the user can touch the base with one finger. The pedestal on which the touch gesture has been input is visually displayed differently (e.g., darker or brighter, etc.) than the pedestal on which the touch gesture has not been input, thereby providing feedback to the user that the touch gesture has been accurately input for the pedestal. In step 2020, when the user inputs a drag gesture based on one finger, a corresponding movement of the object corresponding to the pedestal may be performed on a second reference plane including the pedestal.

Referring to Figure 21, control of an object based on two touch gestures for a pedestal is exemplarily shown. At step 2110, the user can touch the base with two fingers. In step 2120, when the user inputs a drag gesture that changes the angle formed by the two fingers, rotation of the object corresponding to the pedestal may be performed on the second reference plane in response. In step 2130, when the user inputs a drag gesture in which the distance between the two fingers is maintained at a certain level, the object corresponding to the pedestal may be moved on the second reference plane in response.

Referring to Figure 22, control of an object based on two touch gestures for a card and a pedestal is exemplarily shown. In step 2210, the user may touch the base of the object placed in the virtual reality space with one finger and touch the card containing the object with the other finger. In step 2220, if the user inputs a drag gesture for the card while maintaining the touch gesture for the base, the height of the card in the virtual reality space may be controlled correspondingly. The height of objects included in the card can be similarly controlled.

Referring to FIG. 23, control of objects based on two touch gestures for a card and other areas is exemplarily shown. At step 2310, the user may touch the pedestal with one finger and touch a card or other area other than the pedestal with another finger. In step 2320, if the user inputs a drag gesture for another area while maintaining the touch gesture for the pedestal, rotation of the object may be performed in response.

Figures 24 and 25 are examples for explaining operations related to manipulating a card including an object according to an embodiment.

Referring to FIG. 24, objects and cards containing objects can be moved, rotated, and resized within preset boundaries within a virtual reality space. For example, even if the user touches the pedestal with one finger in step 2410 and inputs a drag gesture for the pedestal in step 2420, movement of the object may be limited to within preset boundaries. Likewise, rotation and scaling of objects may also be limited to within preset boundaries.

Referring to FIG. 25, copying and deleting objects can be performed. When a touch gesture 2510 is input to the copy button displayed on the display while an object in the virtual reality space is selected, the selected object (or a card containing the object) is back-projected, thereby creating a copy of the object on the screen plane of the view projection. Copies may be created. Additionally, when the previously described camera manipulation-related operation is performed, the copy 2520 may be displayed separately from the existing object according to the operation. Additionally, there is a delete button on the display, and when a touch gesture is input to the delete button while an object is selected, the selected object can be deleted from the virtual reality space.

Figure 26 is an example to explain a multi-user related operation according to an embodiment.

Referring to FIG. 26, when a plurality of electronic devices are connected to a virtual reality space, view frustums 2610 and 2620 of other electronic devices displayed on the display of one electronic device are shown as an example. A virtual reality space can be accessed by multiple users simultaneously. Each user can view a screen according to the corresponding view frustum through the display of his or her electronic device, and the screen shown in FIG. 26 may be an example displayed on the display of the first electronic device. At this time, the view frustums 2610 and 2620 of the second and third electronic devices may be displayed, and if a sketch is performed on the screen plane of the corresponding view frustum in the second and third electronic devices, the object being sketched may be visualized in real time on the display of the first electronic device. Additionally, the positions and directions of the view frustums 2610 and 2620 of the second and third electronic devices can be visualized in real time based on the camera viewpoints and gaze directions of the second and third electronic devices. Through this, various experiences can be provided to users through interaction between users within the virtual reality space.

27 to 29 are examples for explaining HMD-related operations according to an embodiment.

Referring to FIG. 27, the user can control objects in the virtual reality space while wearing the HMD 2720. The HMD 2720 is worn on the user's head and can display a screen in the virtual reality space according to the direction of the user's head from a viewpoint corresponding to the view frustum in the virtual reality space. The view frustum corresponding to the HMD 2720 may be different from the view frustum corresponding to the electronic device described above. For convenience of explanation, the first view frustum corresponds to the electronic device, and the second view frustum corresponds to the electronic device. It can be explained as corresponding to HMD (2720). While the first view frustum is controlled through one or more touch gestures as described above, the second view frustum may be controlled according to the movement of the head of the user wearing the HMD 2720. However, the embodiment is not limited to this, and depending on the embodiment, the first view frustum and the second view frustum may be the same.

If the screen displayed on the display of the electronic device 2710 is called a first screen, and the screen displayed on the HMD 2720 is called a second screen, the first screen may be displayed on the second screen depending on the direction of the user's head. there is. As will be explained in detail later, if the user's head is facing the electronic device 2710, the display of the electronic device 2710 may be displayed on the second screen displayed on the HMD 2720, and the display of the electronic device 2710 may be displayed on the second screen. The display may display a first screen. To this end, a position difference between the electronic device 2710 and the HMD 2720 may be detected. The position and/or direction of the electronic device 2710 and the HMD 2720 may be detected through a sensor included in the electronic device 2710 or connected to the electronic device 2710. The position and/or direction of the electronic device 2710 may be detected through the module 2730 disposed at a predetermined location of the electronic device 2710, or may be detected based on a marker other than the module 2730. , it is not limited to the above-described example, and the position and/or direction of the electronic device 2710 may be detected without a separate module 2730 or marker.

Referring to FIG. 28, in step 2810, a user not wearing an HMD performs control on an object and/or a first view frustum in the virtual reality space using the first screen displayed on the actual display of the electronic device 2820. An example is shown. A user may perform control over the object and/or the first view frustum based on one or more touch gestures.

Step 2830 shows an example of a second screen displayed on the HMD by a user wearing the HMD. The virtual display 2840 of the electronic device 2820 may be displayed on the second screen. The virtual display 2840 can be synchronized with the actual display of the electronic device 2820. The virtual display 2840 may display the first screen displayed on the actual display of the electronic device 2820.

One or a combination of two or more of the size, position, and direction of the virtual display 2840 of the electronic device 2820 displayed on the second screen may be synchronized with the actual display of the electronic device 2820. The tracked user's hand may be displayed on the second screen. The user can view his or her hand displayed on the second screen and the virtual display 2840, and input one or more touch gestures described above into the virtual display 2840. One or a combination of two or more of the size, position, and orientation of the virtual display 2840 is synchronized with the actual display of the electronic device 2820, and the virtual display 2840 is displayed on the physical display of the electronic device 2820. The screen is displayed, and the virtual display 2840 of the electronic device 2820 and the user's tracked hand are displayed on the second screen, so the user's action of inputting a touch gesture on the virtual display 2840 touches the actual display. Gestures may be input. The user can input a touch gesture on the actual display of the electronic device 2820 without any heterogeneity just by looking at the second screen displayed on the HMD, and through this, various controls based on the touch gesture described above can be performed in the same way. The operations described above can be performed similarly for pen drawings as well as touch gestures.

Referring to FIG. 29, an example of a user wearing an HMD controlling an object using a pinch gesture is shown. In step 2910, the user can hold a card containing an object with the tracked and displayed hand while viewing the second screen displayed on the HMD. In step 2920, the user may move the hand while holding the card, and movement may be performed on the card containing the object. In step 2930, the user may rotate the hand while holding the card, and a corresponding rotation may be performed on the card containing the object. Although not shown in FIG. 29, the user can hold a card containing an object with both hands that are tracked and displayed while viewing the second screen displayed on the HMD. While holding the card with both hands, the user may move and/or rotate both hands, change the distance between the two hands, and, in response, make one or more of the following movements, rotations, and scaling adjustments to the card containing the object. Combinations can be performed.

Figure 30 is a diagram showing a method of operating an electronic device according to an embodiment.

Referring to FIG. 30, in step 3010, the electronic device displays a first screen according to the first view frustum corresponding to the electronic device in the virtual reality space. In step 3020, the electronic device receives control of the virtual reality space and/or an object within the virtual reality space through one or more touch gestures. In response to the case where the user controlling the electronic device wears the HMD on the head, the HMD displays a second screen in the virtual reality space according to the direction of the user's head at a viewpoint corresponding to the second view frustum in the virtual reality space. do. The virtual display of the electronic device displayed on the second screen according to the direction of the head is synchronized with the actual display of the electronic device. When the user performs an action of inputting one or more touch gestures to the virtual display displayed on the second screen, the one or more touch gestures are input to the actual display of the electronic device to control the virtual reality space and/or object.

Since the details described above with reference to FIGS. 1 to 29 are applied to each step shown in FIG. 30, detailed description will be omitted.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (19)

In electronic systems,
Control of a virtual reality space and/or an object within the virtual reality space is input through one or more touch gestures, and a first screen according to a corresponding first view frustum within the virtual reality space is displayed. electronic device for displaying; and
A head mounted display (HMD) that is worn on the user's head and displays a second screen in the virtual reality space according to the direction of the user's head at a viewpoint corresponding to the second view frustum in the virtual reality space.
Including,
The virtual display of the electronic device displayed on the second screen according to the direction of the head corresponds to and is synchronized with the actual display of the electronic device,
When one or more touch gestures for the virtual display displayed on the second screen are detected, the virtual reality space and/or the Control of an object is performed, and the results of the control in the virtual reality space and/or the object are displayed on the actual display of the electronic device and the virtual display displayed on the second screen,
electronic system.
According to paragraph 1,
The virtual display of the electronic device displayed on the second screen displays the first screen displayed on the actual display of the electronic device,
electronic system.
According to paragraph 1,
One or a combination of two or more of the size, position, and orientation of the virtual display of the electronic device displayed on the second screen is synchronized with the actual display of the electronic device,
electronic system.
According to paragraph 1,
The user's tracked hand is displayed on the second screen along with the virtual display of the electronic device, and the user's action of inputting a touch gesture on the virtual display inputs the touch gesture on the actual display,
electronic system.
According to paragraph 1,
The electronic device is
Perform movement and/or rotation of the first view frustum in the virtual reality space based on one or more touch gestures input from the user,
The movement and/or rotation with respect to the first view frustum is
Performed within preset boundaries within the virtual reality space,
electronic system.
According to paragraph 1,
The electronic device is
Performing a sketch on the screen plane of the first view frustum in the virtual reality space based on a pen drawing input to the actual display, and displaying the sketched screen plane as the first screen,
electronic system.
According to clause 6,
The electronic device is
Based on one or more touch gestures for the object created by the sketch, performing one or a combination of two or more of movement, rotation, and resizing of the object on the screen plane,
electronic system.
According to clause 6,
The electronic device is
Determining the projection surface of the object created by the sketch based on the gaze direction of the first view frustum when performing the sketch,
electronic system.
According to clause 8,
The electronic device is
When a creation command for the object is input from the user, the object is created on the projection surface and placed in the virtual reality space,
electronic system.
According to paragraph 1,
The electronic device is
Based on one or more touch gestures on a card containing an object in the virtual reality space, performing one or a combination of two or more of movement, rotation, and scaling of the object on a first reference plane containing the card,
electronic system.
According to paragraph 1,
The electronic device is
Based on one or more touch gestures with respect to a pedestal of a card containing an object in the virtual reality space, performing a movement and/or rotation with respect to the object on a second reference plane containing the pedestal,
The pedestal is
The card is generated parallel to the floor at a position projected on the floor of the virtual reality space,
electronic system.
According to paragraph 1,
The electronic device is
Controlling the height of the card in the virtual reality space based on a second touch gesture for the card while a first touch gesture for the base of the card containing an object in the virtual reality space is input,
electronic system.
According to paragraph 1,
The electronic device is
While a first touch gesture is input to a pedestal of a card containing an object in the virtual reality space, a rotation is performed on the object based on a second touch gesture to an area other than the card or the pedestal,
electronic system.
According to paragraph 1,
One or a combination of two or more of movement, rotation, and size adjustment of the object according to one or more touch gestures input from the user is performed within a preset boundary in the virtual reality space,
electronic system.
According to paragraph 1,
The first screen and/or the second screen
Displaying a third view frustum corresponding to a second electronic device connected to the virtual reality space in the viewpoint direction of the third view frustum at a position corresponding to the second electronic device,
electronic system.
According to clause 15,
The first screen and/or the second screen
In response to a sketch being performed on the second screen plane of the third view frustum based on a pen drawing input to the actual display of the second electronic device, displaying the sketched second screen plane,
electronic system.
According to paragraph 1,
The electronic device is
Performing one or a combination of two or more of movement, rotation, and size adjustment for the object based on one or more pinch gestures for the object placed in the virtual reality space input from the user wearing the HMD,
electronic system.
In a method of operating an electronic device,
An operation of displaying a first screen according to a first view frustum corresponding to the electronic device in a virtual reality space; and
An operation of receiving control of the virtual reality space and/or objects within the virtual reality space using one or more touch gestures.
Including,
In response to the case where the user controlling the electronic device wears the HMD on the head, the HMD is configured to 2 Display the screen,
The virtual display of the electronic device displayed on the second screen according to the direction of the head corresponds to and is synchronized with the actual display of the electronic device,
When one or more touch gestures for the virtual display displayed on the second screen are detected, the virtual reality space and/or the Control of an object is performed, and the results of the control in the virtual reality space and/or the object are displayed on the actual display of the electronic device and the virtual display displayed on the second screen,
How electronic devices work.
A computer-readable recording medium storing a computer program for executing the method of claim 18.

Images