WO2025150937A1 - Method for operating mixed reality telepresence system and mixed reality telepresence system performing same - Google Patents

Abstract

A method for operating a mixed reality telepresence system comprises the steps of: extracting, from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients, a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set; on the basis of the first subspace, the plurality of second subspaces, and an optimization function, generating a plurality of alignment spaces in which the plurality of second subspaces overlap the first subspace; determining a plurality of first positions on the plurality of alignment spaces; and on the basis of the plurality of first positions, setting a plurality of first interactable areas in the first virtual space.

Description

Method of operation of a mixed reality telepresence system and a mixed reality telepresence system performing the same

The present invention relates to a mixed reality system, and more particularly, to an operating method of a mixed reality telepresence system and a mixed reality telepresence system performing the operating method.

Recently, mixed reality (MR) telepresence (or remote collaboration) systems that connect people through augmented reality (AR) and virtual reality (VR) technologies have been continuously receiving attention. In MR telepresence, users can achieve a high level of copresence with other remote users as avatars placed in a mutual workspace. This space is mostly created based on the AR space of the host user, and the remote users are virtually transported while wearing VR HMD (head-mounted display). In fact, most research on MR telepresence in the past decade has focused on the technical asymmetric system involving the AR host and VR client.

One of the key challenges of asymmetric MR telepresence is to understand each user’s heterogeneous space and generate an optimized interactivity zone where the AR host and VR client can coexist effectively. Past methods have tried to find the maximum intersection area of relevant subspaces, but as the number of subspaces increases, their size decreases, which imposes more restrictions on the user’s movement.

One object of the present invention is to provide a method of operating a mixed reality telepresence system capable of providing excellent spatial consistency according to a collaborative context.

One object of the present invention is to provide a mixed reality telepresence system that performs the operating method of the mixed reality telepresence system.

In order to achieve the above object, the operating method of the mixed reality telepresence system according to embodiments of the present invention includes the steps of extracting a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients, generating a plurality of alignment spaces overlapping the plurality of second subspaces with the first subspace based on the first subspace, the plurality of second subspaces, and an optimization function, determining a plurality of first positions on the plurality of alignment spaces, and setting a plurality of first interactable areas in the first virtual space based on the plurality of first positions.

In order to achieve the above object, a mixed reality telepresence system according to embodiments of the present invention includes a subspace extraction unit which extracts a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients, a subspace alignment unit which receives the first subspace and the plurality of second subspaces from the subspace extraction unit, and generates a plurality of alignment spaces overlapping the plurality of second subspaces with the first subspace based on the first subspace, the plurality of second subspaces, and an optimization function, and a subspace setting unit which receives the plurality of alignment spaces from the subspace alignment unit, determines a plurality of first positions on the plurality of alignment spaces, and sets a plurality of first interactable areas in the first virtual space based on the plurality of first positions.

In order to achieve the above object, the operating method of the mixed reality telepresence system according to embodiments of the present invention comprises the steps of: obtaining initial subspaces from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients; calculating a target object from the initial subspaces; obtaining a first subspace corresponding to the first virtual space and a plurality of second subspaces corresponding to the second virtual space based on the initial subspaces and the target object; determining an optimization function based on a collaboration context; setting a second position and a second angle for each of the plurality of second subspaces based on the first subspace, the plurality of second subspaces, and the optimization function; obtaining a plurality of alignment spaces overlapping the plurality of second subspaces with the first subspace based on the second position and the second angle; determining a plurality of first positions on the plurality of alignment spaces; setting different first to fourth markers in the form of lines at the plurality of first positions; and moving the first to fourth markers until a first condition is satisfied. The step of moving in the first to fourth directions, respectively, and the step of generating polygons formed by extending the first to fourth markers as the plurality of first interactable areas.

In the operation method of the mixed reality telepresence system according to the embodiments of the present invention as described above, interactable areas can be derived by using data that overlaps (or aligns) the subspace of each of the plurality of clients with the subspace of the host. When the interactable areas are allocated to each of the plurality of clients, excellent spatial consistency according to the collaboration context can be provided regardless of an increase in the number of clients or heterogeneity with the host space.

FIG. 1 is a flowchart illustrating an operation method of a mixed reality telepresence system according to embodiments of the present invention.

FIG. 2 is a block diagram illustrating a mixed reality telepresence system according to embodiments of the present invention.

FIG. 3 is a flowchart illustrating an example of extracting a first subspace and a plurality of second subspaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

FIG. 4 is a diagram for explaining acquisition of initial subspaces of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

FIG. 5 is a flowchart illustrating an example of target object extraction from initial subspaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

FIGS. 6, 7, 8 and 9 are diagrams for explaining target object extraction from initial subspaces of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

FIG. 10 is a flowchart illustrating an example of generating multiple alignment spaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

FIGS. 11 and 12 are drawings for explaining the creation of multiple alignment spaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

FIG. 13 is a flowchart illustrating an example of determining a plurality of first positions in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

FIGS. 14, 15, 16 and 17 are drawings for explaining determination of a plurality of first positions of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

FIG. 18 is a flowchart illustrating an example of setting a plurality of first interactable areas in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

FIGS. 19, 20, 21, 22, 23 and 24 are drawings for explaining setting of a plurality of first interactable areas of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Hereinafter, with reference to the attached drawings, a preferred embodiment of the present invention will be described in more detail. The same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

FIG. 1 is a flowchart illustrating an operation method of a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIG. 1, the operation method of the mixed reality telepresence system according to the embodiments of the present invention can be applied to a telepresence system in which a host and a plurality of clients remotely collaborate using an augmented reality (AR) device and/or a virtual reality (VR) device. For example, the mixed reality telepresence system according to the embodiments of the present invention can refer to a system in which a plurality of clients are virtually transported while wearing VR devices based on the AR space of the host.

A first subspace and a plurality of second subspaces are extracted from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients (step S100). For example, the first virtual space and the plurality of second virtual spaces may represent data collected through an AR/VR device. For example, the first subspace may represent a part of the first virtual space data, and the plurality of second subspaces may represent a part of the data of the plurality of second virtual spaces.

For example, the plurality of clients may be implemented as three-dimensional avatars in the first virtual space of the host. For example, the host may visually display the avatars of the plurality of clients on the first virtual space. For example, the plurality of clients may visually display a portion of the first virtual space implemented on the plurality of second virtual spaces.

For example, the host may be one or more. If the host is multiple, it may indicate a case where there are multiple hosts sharing the first virtual space. That is, the host may be multiple, but the first virtual space may be unique.

For example, a target object may be set in the first subspace and the plurality of second subspaces. For example, the target object may represent an interactable object. For example, the interactable object may represent an object that the host or the plurality of clients can directly feel through the five senses. For example, the target object may represent a table, a wall, a floor, etc. Specific examples of the first subspace and the plurality of second subspaces will be described later with reference to FIGS. 3 to 9.

Based on a first subspace, a plurality of second subspaces, and an optimization function, a plurality of alignment spaces are generated (step S200). For example, the optimization function may be used when aligning or overlapping the plurality of second subspaces with the first subspace. For example, the optimization function may be determined based on a collaboration context. For example, when a remote conference is conducted at a table, the optimization function may be set so that the table existing in the plurality of second subspaces and the table existing in the first subspace overlap as much as possible, that is, have the largest possible overlapping area. For example, the plurality of alignment spaces may represent data that overlaps the data of the plurality of second subspaces with the data of the first subspace. A specific process of generating the plurality of alignment spaces will be described later with reference to FIGS. 10 to 12.

A plurality of first positions on a plurality of alignment spaces are determined (step S300). For example, the plurality of first positions may represent initial positions of the plurality of clients. For example, the initial positions of the plurality of clients may represent positions of the plurality of clients at the initial operation of the mixed reality telepresence system. For example, the plurality of first positions may include positions of the host as well as positions of the plurality of clients. For example, the plurality of first positions may be used to set first interactable areas. A specific process of determining the plurality of first positions will be described below with reference to FIGS. 13 to 17.

Based on a plurality of first locations, a plurality of first interactable areas are set in a first virtual space (step S400). For example, the plurality of first interactable areas may include different interactable areas of each of the plurality of clients. For example, each of the plurality of clients may be located on the first virtual space of the host within a range of the different interactable areas. A specific process of setting a plurality of first interactable areas will be described later with reference to FIGS. 18 to 24.

In the operating method of the mixed reality telepresence system according to the embodiments of the present invention as described above, a plurality of first interactable areas can be derived by using data that overlaps (or aligns) a plurality of second subspaces of each of a plurality of clients with the first subspace of the host. When the plurality of first interactable areas are allocated to each of the plurality of clients, excellent spatial consistency according to the collaboration context can be provided regardless of an increase in the number of clients or heterogeneity with the host space.

FIG. 2 is a block diagram illustrating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIG. 2, the mixed reality telepresence system (1000) includes a partial space extraction unit (100), a partial space alignment unit (200), and a partial space setting unit (300). As described above, the mixed reality telepresence system (1000) may represent a system in which a plurality of clients are virtually transported while wearing VR devices based on the AR space of the host.

The subspace extraction unit (100) extracts a first subspace (SS1) in which a target object is set and a plurality of second subspaces (SS2) in which the target object is set from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients. For example, the subspace extraction unit (100) can receive data of the first virtual space and the plurality of second virtual spaces from an AR/VR device. For example, the subspace extraction unit (100) can obtain initial subspaces from the first virtual space and the plurality of second virtual spaces. For example, the subspace extraction unit (100) can derive the target object from the initial subspaces. For example, the subspace extraction unit (100) can obtain the first subspace (SS1) and a plurality of second subspaces (SS2) by setting the target object in the initial subspaces. For example, the subspace extraction unit (100) can perform step S100 of FIG. 1. For example, the subspace extraction unit (100) can perform steps S110, S120, and S130 of FIG. 3, which will be described later. For example, the subspace extraction unit (100) can perform steps S121 and S122 of FIG. 5, which will be described later.

The subspace alignment unit (200) receives a first subspace (SS1) and a plurality of second subspaces (SS2) from the subspace extraction unit (100). In addition, the subspace alignment unit (200) generates a plurality of alignment spaces (AS) that overlap the first subspace (SS1) with the plurality of second subspaces (SS2) based on the first subspace (SS1), the plurality of second subspaces (SS2), and the optimization function. For example, the subspace alignment unit (200) can determine the optimization function based on the collaboration context. For example, the subspace alignment unit (200) can set a second position and a second angle for each of the plurality of second subspaces (SS2) based on the first subspace (SS1), the plurality of second subspaces (SS2), and the optimization function, and can obtain a plurality of alignment spaces based on the second position and the second angle. For example, the subspace alignment unit (200) can perform step S200 of Fig. 1. For example, the subspace alignment unit (200) can perform steps S210, S220, and S230 of Fig. 10, which will be described later.

The partial space setting unit (300) receives a plurality of alignment spaces (AS) from the partial space alignment unit (200). In addition, the partial space setting unit (300) determines a plurality of first positions on the plurality of alignment spaces (AS), and sets a plurality of first interactable areas in the first virtual space based on the plurality of first positions. For example, the partial space setting unit (300) can output the first virtual space in which the plurality of first interactable areas are set. For example, the mixed reality telepresence system can be operated based on the first virtual space in which the plurality of first interactable areas are set. For example, the partial space setting unit (300) can perform steps S300 and S400 of FIG. 1. For example, the partial space setting unit (300) can perform steps S310, S320, and S330 of FIG. 13, which will be described later. For example, the partial space setting unit (300) can perform steps S410, S420, and S430 of FIG. 18, which will be described later.

FIG. 3 is a flowchart illustrating an example of extracting a first subspace and a plurality of second subspaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIG. 3, steps S110, S120, and S130 may represent an example of step S100 of FIG. 1.

Initial subspaces can be acquired from a first virtual space and a plurality of second virtual spaces (step S110). For example, the initial subspaces can include information about a floor, first objects that can be interacted with, second objects on which a host and a plurality of clients can sit, and third objects representing obstacles. For example, the initial subspaces can represent data that expresses the floor, the first objects, the second objects, and the third objects in the form of polygons.

The target object can be derived from the initial subspaces (step S120). For example, the target object can be selected as one of the first objects. For example, when selecting the target object among the first objects, a scene graph can be used. The specific process of generating the target object will be described later with reference to FIGS. 5 to 9.

A first subspace and a plurality of second subspaces can be obtained based on the initial subspaces and the target object (step S130). For example, the first subspace and the plurality of second subspaces can represent data that sets the target object in the initial subspaces.

FIG. 4 is a diagram for explaining acquisition of initial subspaces of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIG. 4, initial subspaces (PSS) can be obtained from a first virtual space (VS1) and a plurality of second virtual spaces (VS2). For example, the initial subspaces (PSS) can include a first initial subspace (PSS1) corresponding to the first virtual space (VS1) and a plurality of second initial subspaces (PSS2) corresponding to the plurality of second virtual spaces (VS2). For example, the initial subspaces (PSS) can include information about a floor (FL), first objects (O1), second objects (O2), and third objects (O3). For example, the initial subspaces (PSS) can represent data that expresses the floor (FL), the first objects (O1), the second objects (O2), and the third objects (O3) in the form of rectangles, respectively.

FIG. 5 is a flowchart illustrating an example of target object extraction from initial subspaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIG. 5, steps S121 and S122 may represent examples of step S120 of FIG. 3.

A target object can be selected from among the first objects on the first initial subspace (step S121). For example, in the first initial subspace (PSS1) of FIG. 4, there is one first object, but there may be multiple first objects on the first initial subspace. For example, if there is one first object on the first initial subspace, the first object can be automatically selected as the target object. For example, if there are multiple first objects on the first initial subspace, the target object of the first initial subspace can be selected by a user (e.g., host).

Using the target object and the scene graph on the first initial subspace, the target object can be determined among the first objects on the plurality of second initial subspaces (step S122). For example, the scene graph can represent a data structure including a set of nodes of a graph or tree structure. For example, the scene graph can be used to infer not only objects but also relationships between objects for an input image. For example, the scene graph can classify objects by instance from the input image and express relationships using a connection graph. A specific process of selecting a target object using the scene graph will be described later using FIG. 7.

FIGS. 4, 6, 7, 8 and 9 are drawings for explaining target object extraction from initial subspaces of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIGS. 6, 7, 8 and 9, a process of obtaining a first subspace (SS1) and a plurality of second subspaces (SS2) from a first initial subspace (PSS1) and a plurality of second initial subspaces (PSS2) can be illustrated.

As illustrated in Fig. 6, the target object (TO) may be selected as one of the first objects (O1) on the first initial subspace (PSS1). For example, since there is one first object (O1) in the first initial subspace (PSS1), the target object (TO) may be selected as one first object (O1).

As illustrated in FIG. 7, a target object (TO) of one of the first objects (O1) on a plurality of second initial subspaces (PSS2) can be determined using a new graph. Hereinafter, a second-first initial subspace (PSS2_1) obtained from a virtual space of a first client among a plurality of clients will be described as an example. For example, the second-first initial subspace (PSS2_1) can include two first objects (O1_1, O1_2). For example, a first new graph (SG1) can be generated for a target object (TO_H) of the first initial subspace (PSS1). For example, the first new graph (SG1) can generate nodes (SG_TO, SG_O2) for the target object (TO_H) and surrounding objects, and connect the nodes (SG_TO, SG_O2). For example, the first scene graph (SG1) may include relationship information between the target object (TO_H) and the second objects (O2). Similarly, scene graphs may also be generated for the first objects (O1_1, O1_2) of the 2-1 initial subspace (PSS2_1). Thereafter, the scene graphs of the first scene graph (SG1) and the first objects (O1_1, O1_2) may be compared to select the first object that most closely matches the target object (TO_H) by considering the size of the first objects (O1_1, O1_2) as well as the relationship with surrounding objects.

As illustrated in Fig. 8, through comparison of the new graphs, the first object (O1_1) among the first objects (O1_1, O1_2) of Fig. 7 can be selected as the target object. For example, the remaining first objects that were not selected as the target objects can be treated as third objects representing obstacles.

As illustrated in FIG. 9, a first subspace (SS1) in which one target object (TO) is set in the first initial subspace (PSS1) of FIG. 4 can be obtained, and a plurality of second subspaces (SS2) in which one target object (TO) is set in the plurality of second initial subspaces (PSS2) of FIG. 4 can be obtained.

FIG. 10 is a flowchart illustrating an example of generating multiple alignment spaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIG. 10, steps S210, S220 and S230 may represent an example of step S200 of FIG. 1.

Based on the collaboration context, an optimization function can be determined (step S210). For example, if the target object is a table, the optimization function can be determined so that the tables existing in the plurality of second subspaces and the tables existing in the first subspace overlap as much as possible, i.e., so that they have the widest possible overlapping area.

Based on the first subspace, the plurality of second subspaces, and the optimization function, a second position and a second angle can be set for each of the plurality of second subspaces (step S220). For example, the second position and the second angle can be calculated by the optimization function so that the target object of the first subspace and the target objects of the plurality of second subspaces overlap as much as possible. For example, the second position can represent a relative position of the plurality of second subspaces in relation to the first subspace. For example, the second angle can represent a relative rotation degree of the plurality of second subspaces in relation to the first subspace.

Based on the second position and the second angle, a plurality of alignment spaces can be obtained (step S230). For example, the plurality of alignment spaces can include the first partial space data and a plurality of second partial space data that are moved by the second position and rotated by the second angle.

FIGS. 11 and 12 are drawings for explaining the creation of multiple alignment spaces in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIGS. 11 and 12, a process of obtaining a plurality of aligned spaces (AS1, AS2, AS3, AS4) overlapping a plurality of second subspaces (SS2_1, SS2_2, SS2_3, SS2_4) with a first subspace (SS1) based on second positions (L2_1, L2_2, L2_3, L2_4) and second angles (A2_1, A2_2, A2_3, A2_4) can be illustrated.

As illustrated in FIG. 11, the second positions (L2_1, L2_2, L2_3, L2_4) and the second angles (A2_1, A2_2, A2_3, A2_4) can be calculated based on the optimization function. For example, the second angle (A2_1) of the 2-1st subspace (SS2_1) can be calculated as 0°, the second angle (A2_2) of the 2-2nd subspace (SS2_2) can be calculated as 90°, the second angle (A2_3) of the 2-3rd subspace (SS2_3) can be calculated as 0°, and the second angle (A2_4) of the 2-4th subspace (SS2_4) can be calculated as 90°.

As illustrated in FIG. 12, based on the second positions (L2_1, L2_2, L2_3, L2_4) and the second angles (A2_1, A2_2, A2_3, A2_4) of FIG. 11, a plurality of alignment spaces (AS1, AS2, AS3, AS4) can be obtained. For example, when the plurality of clients include the first to fourth clients, the first alignment space (AS1) can represent data overlapping (or aligned) the subspaces of the host and the first client. Similarly, the second alignment space (AS2) can represent data overlapping (or aligned) the subspaces of the host and the second client, the third alignment space (AS3) can represent data overlapping (or aligned) the subspaces of the host and the third client, and the fourth alignment space (AS4) can represent data overlapping (or aligned) the subspaces of the host and the fourth client.

FIG. 13 is a flowchart illustrating an example of determining a plurality of first positions in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIG. 13, steps S310, S320 and S330 may represent an example of step S300 of FIG. 1.

A plurality of first sample positions of a plurality of clients can be set to be spaced apart from a first overlap area by a first distance (step S310). For example, the first overlap area can represent an area where a target object of a first subspace and target objects of a plurality of second subspaces overlap. For example, the first overlap area can be included in a plurality of alignment spaces. For example, the first distance can be set so that the host or the plurality of clients can move around the target object without being restricted by the target object. For example, when the target object is a table, the first distance can be set to an extent (e.g., 20 cm) that a person can move around the table without being restricted by the table. For example, the plurality of first sample positions can be expressed in the form of a plurality of dots around the first overlap area.

A plurality of second sample locations can be selected from a plurality of first sample locations based on obstacle information, body information of a plurality of clients, etc. (step S320). For example, the obstacle information can represent information related to third objects on the plurality of aligned spaces. For example, the body information of the plurality of clients can include shoulder width, stride, etc. of the plurality of clients. For example, the plurality of second sample locations can be obtained by removing some of the plurality of first sample locations. For example, locations overlapping with a location of an obstacle and locations spaced apart from the obstacle by a distance of less than a shoulder width of the client can be excluded from the plurality of first sample locations.

Based on the proxymix between the second sample locations and the plurality of clients, a plurality of first locations can be obtained (step S330). For example, the proxymix can represent distances between the plurality of clients. For example, if the plurality of clients include the first, second, and third clients, the plurality of first locations can be obtained by considering all of the distances between the first client and the second client, the distances between the second client and the third client, and the distances between the third client and the first client.

FIGS. 14, 15, 16 and 17 are drawings for explaining determination of a plurality of first positions of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIGS. 14, 15, 16 and 17, a process of determining a plurality of first positions (PL1) on a plurality of alignment spaces (AS1, AS2, AS3, AS4) can be illustrated. Hereinafter, for convenience of explanation, it is assumed that the plurality of clients include first to fourth clients, and the plurality of alignment spaces (AS1, AS2, AS3, AS4) include a first alignment space (AS1), a second alignment space (AS2), a third alignment space (AS3), and a fourth alignment space (AS4).

As illustrated in FIG. 14, a plurality of first sample locations (PSL1) of a plurality of clients can be set to be spaced apart from a first overlap area (OA1) by a first distance (DS1). For example, the first overlap area (OA1) can represent an area where a target object of a first subspace and target objects of a plurality of second subspaces overlap.

As illustrated in FIG. 15, a plurality of second sample locations (PSL2) can be obtained by excluding locations that overlap with the location of an obstacle and locations that are spaced apart from the obstacle by a distance less than the shoulder width of the client among a plurality of first sample locations (PSL1).

As illustrated in FIG. 16, a plurality of first positions (PL1) may be set considering the distances between the first to fourth clients, respectively. For example, the plurality of first positions (PL1) may include a first position (PL1_1) of the first client, a first position (PL1_2) of the second client, a first position (PL1_3) of the third client, and a first position (PL1_4) of the fourth client. In this case, for the convenience of distance comparison between the first to fourth clients, the plurality of alignment spaces (AS1, AS2, AS3, AS4) may be integrated into one integrated alignment space (TAS).

As illustrated in FIG. 17, in addition to the plurality of first positions (PL1), the plurality of host positions (PHL1) may also be set in the integrated alignment space (TAS) or the plurality of alignment spaces (AS1, AS2, AS3, AS4). As described above with reference to FIG. 1, the host may be plural, and the plurality of host positions (PHL1) may include positions (HL1, HL2, HL3) of the first to third hosts. However, the plurality of host positions (PHL1) may be set at any time after the plurality of first positions (PL1) are set.

FIG. 18 is a flowchart illustrating an example of setting a plurality of first interactable areas in a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIG. 18, steps S410, S420, and S430 may represent an example of step S400 of FIG. 1.

Different first to fourth markers in the form of line segments can be set at a plurality of first locations (step S410). For example, the first to fourth markers can represent an example of a method for producing first interactable areas. For example, the first to fourth markers can be arranged to surround a plurality of first locations. For example, the first to fourth markers can be implemented in the form of line segments or rectangular bars.

The first to fourth markers can be moved in the first to fourth directions, respectively, until a first condition is satisfied (step S420). For example, the first condition may include that the first to fourth markers do not come into contact with third objects or a target object. For example, the first condition may include that the first to fourth markers do not come into contact with floor boundaries of the plurality of alignment spaces. For example, the first to fourth markers can be arranged in a square shape at the beginning of the operation of step S420, and the first to fourth markers can be moved in the up, down, left, and right directions, respectively, until the first condition is satisfied.

The polygons formed by extending the first to fourth markers can be calculated as a plurality of first interactable areas (step S430). For example, the positions of the first to fourth markers at the beginning of the operation in step S420 and the positions after the movement can be different. For example, when four of the first to fourth markers are extended, a square can be formed. Therefore, when the first to fourth markers are used, the plurality of first interactable areas can be calculated in the shape of a square.

FIGS. 19, 20, 21, 22, 23 and 24 are drawings for explaining setting of a plurality of first interactable areas of a method of operating a mixed reality telepresence system according to embodiments of the present invention.

Referring to FIGS. 19, 20, 21, 22, 23 and 24, a process of obtaining a plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4) using first to fourth markers (M1, M2, M3, M4) can be illustrated.

Hereinafter, it will be described assuming that the plurality of clients include first to fourth clients, the plurality of alignment spaces include first to fourth alignment spaces, and the plurality of first interactable areas include a first-first interactable area (ISS1_1), a first-second interactable area (ISS1_2), a first-third interactable area (ISS1_3), and a first-fourth interactable area (ISS1_4). In addition, although the case of the first alignment space (AS1) is described below as a representative example, interactable areas can be produced in the second to fourth alignment spaces as well, as will be described below with reference to FIGS. 19 to 21.

As illustrated in FIG. 19, the first alignment space (AS1) may represent a space that overlaps the second subspace of the first client with the first subspace of the host. For example, the first to fourth markers (M1, M2, M3, and M4) may be arranged to surround the first location (PL1_1) of the first client. For example, the first marker (M1) may move in the first direction (D1), the second marker (M2) may move in the second direction (D2), the third marker (M3) may move in the third direction (D3), and the fourth marker (M4) may move in the fourth direction (D4).

As illustrated in FIG. 20, the first to fourth markers (M1, M2, M3, M4) can move in the first to fourth directions (D1, D2, D3, D4) respectively until the first condition is satisfied. For example, the first marker (M1) can move until it contacts the target object. For example, the second to fourth markers (M2, M3, M4) can move until it contacts the floor boundary line of the first alignment space (AS1).

As illustrated in Fig. 21, a polygon formed by extending the first to fourth markers (M1, M2, M3, M4) can be obtained as the first-first interaction possible area (ISS1_1).

As illustrated in FIG. 22, the first-second interactable area (ISS1_2), the first-third interactable area (ISS1_3), and the first-fourth interactable area (ISS1_4) can be obtained as described above with reference to FIGS. 19 to 21.

As illustrated in FIG. 23, according to an embodiment, each of the plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4) may further include the first overlapping areas described above with reference to FIG. 14.

As illustrated in FIG. 24, a plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4) may be finally set in the first virtual space (VS1) of the host. For example, the first virtual space (VS1) in which the plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4) are set may be used for the operation of the mixed reality telepresence system. For example, the first virtual space may include all data related to the plurality of first interactable areas (ISS1_1, ISS1_2, ISS1_3, ISS1_4). For example, data related to the first-first interactable area (ISS1_1) of the first client may be provided only to the first client among the plurality of clients. Similarly, data related to the 1st-2nd, 1st-3rd and 1st-4th interactable areas (ISS1_2, ISS1_3, ISS1_4) may be provided only to the 2nd, 3rd and 4th clients, respectively.

Additionally, when a client is added, steps S100 to S400 of FIG. 1 may be performed again so that a plurality of new first interactable areas can be set in the first virtual space (VS1).

Embodiments of the present invention can be usefully utilized in any electronic device and system including a storage device. For example, embodiments of the present invention can be more usefully applied to electronic systems such as PCs (Personal Computers), server computers, data centers, workstations, laptops, cellular phones, smart phones, MP3 players, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), digital TVs, digital cameras, portable game consoles, navigation devices, wearable devices, IoT (Internet of Things) devices, IoE (Internet of Everything) devices, e-books, VR (Virtual Reality) devices, AR (Augmented Reality) devices, drones, automotives, and the like. Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims (12)

A step of extracting a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients; A step of generating a plurality of alignment spaces overlapping the plurality of second subspaces with the first subspace based on the first subspace, the plurality of second subspaces, and the optimization function; a step of determining a plurality of first positions on the plurality of alignment spaces; and A method of operating a mixed reality telepresence system, comprising the step of setting a plurality of first interactable areas in the first virtual space based on the plurality of first locations. In the first paragraph, the step of extracting the first subspace in which the target object is set and the plurality of second subspaces in which the target object is set, A step of obtaining initial subspaces from the first virtual space and the plurality of second virtual spaces; a step of generating the target object from the initial subspaces; and A method of operating a mixed reality telepresence system, characterized by comprising a step of obtaining the first subspace and the plurality of second subspaces based on the initial subspaces and the target object. In the second paragraph, A method of operating a mixed reality telepresence system, wherein the initial subspaces include information about a floor, first objects that can be interacted with, second objects on which the host and the plurality of clients can sit, and third objects representing obstacles. In the third paragraph, The above initial subspaces include a first initial subspace corresponding to the first virtual space and a plurality of second initial subspaces corresponding to the plurality of second virtual spaces, The step of generating the target object from the above initial subspaces is: A step of selecting the target object among the first objects in the first initial subspace; and A method of operating a mixed reality telepresence system, characterized by comprising a step of determining a target object among the first objects on the plurality of second initial subspaces by using the target object and scene graph on the first initial subspace. In the first paragraph, The step of generating the above multiple alignment spaces is: A step of determining an optimization function based on the collaboration context; A step of setting a second position and a second angle for each of the plurality of second subspaces based on the first subspace, the plurality of second subspaces, and the optimization function; and A method of operating a mixed reality telepresence system, characterized by comprising a step of obtaining the plurality of alignment spaces based on the second position and the second angle. In paragraph 5, A method of operating a mixed reality telepresence system, characterized in that the above optimization function is set such that an area in which the target object of the first subspace and the target objects of the plurality of second subspaces overlap is maximized. In the first paragraph, A method of operating a mixed reality telepresence system, characterized in that the plurality of alignment spaces include a first overlap area indicating an area where the target object of the first subspace and the target objects of the plurality of second subspaces overlap. In Article 7, The step of determining the plurality of first positions is: A step of setting a plurality of first sample locations of the plurality of clients to be spaced apart from the first overlapping area by a first distance; A step of selecting a plurality of second sample locations among the plurality of first sample locations based on obstacle information, body information of the plurality of clients, etc.; and A method of operating a mixed reality telepresence system, characterized by comprising the step of obtaining the plurality of first locations based on proxemics between the second sample locations and the plurality of clients. In the first paragraph, The step of setting the above plurality of first interactive areas is: A step of setting different first to fourth markers in the form of lines at the plurality of first locations; and A method of operating a mixed reality telepresence system, characterized by comprising a step of acquiring a plurality of first interactable areas based on the first to fourth markers. In Article 9, The step of acquiring the above plurality of first interactable areas is: A step of moving the first to fourth markers in the first to fourth directions respectively until the first condition is satisfied; and A method of operating a mixed reality telepresence system, characterized by comprising a step of generating polygons formed by extending the first to fourth markers as the plurality of first interactable areas. A subspace extraction unit that extracts a first subspace in which a target object is set and a plurality of second subspaces in which the target object is set from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients; A subspace alignment unit that receives the first subspace and the plurality of second subspaces from the subspace extraction unit, and generates a plurality of alignment spaces overlapping the plurality of second subspaces with the first subspace based on the first subspace, the plurality of second subspaces, and an optimization function; and A mixed reality telepresence system comprising a subspace setting unit which receives a plurality of alignment spaces from the subspace alignment unit, determines a plurality of first positions on the plurality of alignment spaces, and sets a plurality of first interactable areas in the first virtual space based on the plurality of first positions. A step of obtaining initial subspaces from a first virtual space of a host and a plurality of second virtual spaces of a plurality of clients; A step of generating target objects from the above initial subspaces; A step of obtaining a first subspace corresponding to the first virtual space and a plurality of second subspaces corresponding to the second virtual space based on the initial subspaces and the target object; A step of determining an optimization function based on the collaboration context; A step of setting a second position and a second angle for each of the plurality of second subspaces based on the first subspace, the plurality of second subspaces, and the optimization function; A step of obtaining a plurality of alignment spaces overlapping the plurality of second sub-spaces with the first sub-space based on the second position and the second angle; A step of determining a plurality of first positions on the plurality of alignment spaces; A step of setting different first to fourth markers in the form of lines at the plurality of first locations; A step of moving the first to fourth markers in the first to fourth directions respectively until the first condition is satisfied; and A method of operating a mixed reality telepresence system, comprising the step of generating polygons formed by extending the first to fourth markers as the plurality of first interactable areas.