KR20220094988A - Method and apparatus for providing augmented reality, virtual reality, mixed reality and extended reality services - Google Patents

Abstract

The present disclosure relates to an apparatus and method for providing augmented reality, virtual reality, mixed reality, and extended reality services, which allow service users to experience more realistic augmented reality by rendering virtual objects based on information on objects existing in the surrounding environment. According to one embodiment, the method includes the steps of: a server's receiving a streaming request and first user information from an electronic device; identifying information about a type of the electronic device and a user's surrounding environmental space based on the first user information; identifying a relationship between multiple objects existing in a first virtual environment space; selecting one or more objects from among the multiple objects based on the information about the type of the electronic device and the user's surrounding environment space, and the relationship between the multiple objects; and transmitting image data about the one or more selected objects to the electronic device.

Description

Apparatus and method for providing augmented reality, virtual reality, mixed reality and extended reality services

The present disclosure relates to an apparatus and method for providing augmented reality, virtual reality, mixed reality and extended reality services.

Recently, as electronic devices capable of providing augmented reality (AR) and virtual reality (VR) services have been developed, interest in technologies (plans) that can utilize them is increasing. Furthermore, with the introduction of 5G mobile communication technology, technologies capable of realizing and utilizing mixed reality (MR) and extended reality (XR) are also being studied.

However, in the case of the current extended reality technology, since the module for the augmented reality service and the module for the virtual reality service exist separately in the server, depending on the type of device (eg, augmented reality device or virtual reality device, etc.) There is a problem that the service that can be supported is limited.

In addition, since the current augmented reality service is provided based on a global positioning system (GPS), there is a limitation that a specific object is rendered to the same location for all users regardless of the user's surrounding environment space.

Based on the above discussion, the present disclosure provides apparatus and methods for providing augmented reality, virtual reality, mixed reality and extended reality services.

According to an embodiment of the present disclosure, the server,

a memory storing one or more instructions;

at least one processor executing the one or more instructions;

The at least one processor by executing the one or more instructions,

Receive a streaming request and first user information from the electronic device,

based on the first user information, identify information about a type of the electronic device and a user's surrounding environment space,

Identifies a relationship between a plurality of objects existing in the first virtual environment space,

selecting one or more objects from among the plurality of objects based on the type of the electronic device, information about the user's surrounding environment space, and a relationship between the plurality of objects,

Image data regarding the one or more selected objects may be transmitted to the electronic device.

The at least one processor may determine information about the first virtual environment space based on media content or information about the user's surrounding environment space.

When selecting one or more objects from among the plurality of objects, if the electronic device is an augmented reality device, the at least one processor is configured to: at least one object existing in the user's surrounding environment space and a relationship between the plurality of objects can be selected based on

When one or more objects are selected from among the plurality of objects, if the electronic device is a virtual reality device, the at least one processor is configured to provide information about a second virtual environment space rendered by the electronic device and information about the space between the plurality of objects. You can choose based on your relationship.

When the at least one processor identifies a relationship between the plurality of objects existing in the first virtual environment space, the plurality of objects are selected based on at least one of a contact area between the plurality of objects and a center-to-center distance. relationship between them can be identified.

When identifying the relationship between the plurality of objects existing in the first virtual environment space, the at least one processor may identify the relationship between the plurality of objects based on semantic reasoning.

The information on the relationship between the plurality of objects may include information on the position relationship between the plurality of objects.

When the at least one processor selects one or more objects from among the plurality of objects, at least one object existing in the environment space surrounding the user or at least one object present in the second virtual environment space rendered by the electronic device A first object may be selected from among the objects, and a second object related to the first object may be selected based on the relationship between the plurality of objects. Also, the at least one processor may select a third object related to the second object based on the relationship between the plurality of objects.

The at least one processor may receive second user information from the electronic device, and the second user information may include information about the user's changed surrounding environment space.

According to an embodiment of the present disclosure, the method performed by the server,

Receiving a streaming request and first user information from the electronic device;

identifying information about a type of the electronic device and a user's surrounding environment space based on the first user information;

identifying a relationship between a plurality of objects existing in a first virtual environment space;

selecting one or more objects from among the plurality of objects based on the type of the electronic device, information about the user's surrounding environment space, and a relationship between the plurality of objects; and

The method may include transmitting image data regarding the one or more selected objects to the electronic device.

The method performed by the server may further include determining information about the first virtual environment space based on media content or information about the user's surrounding environment space.

The selecting of one or more objects from among the plurality of objects may include selecting, when the electronic device is an augmented reality device, based on at least one object existing in a space surrounding the user and a relationship between the plurality of objects. may include the step of

The selecting of one or more objects from among the plurality of objects may include, when the electronic device is a virtual reality device, based on information about a second virtual environment space rendered by the electronic device and a relationship between the plurality of objects. It may include the step of selecting.

The identifying of the relationship between the plurality of objects existing in the first virtual environment space may include: identifying the relationship between the plurality of objects based on at least one of a contact area between the plurality of objects and a distance between centers; may include

The identifying of the relationship between the plurality of objects existing in the first virtual environment space may include identifying the relationship between the plurality of objects based on semantic inference.

The information on the relationship between the plurality of objects may include information on the positional relationship between the plurality of objects.

The selecting of the one or more objects from among the plurality of objects may include: a first among at least one object existing in an environment space surrounding the user or at least one object present in a second virtual environment space rendered by the electronic device selecting an object; and selecting a second object having a relationship with the first object based on the relationship between the plurality of objects. In addition, the selecting of one or more objects from among the plurality of objects may further include selecting a third object having a relationship with the second object based on a relationship between the plurality of objects.

The method performed by the server may further include receiving second user information from the electronic device, wherein the second user information may include information about the user's changed surrounding environment space.

According to an embodiment of the present disclosure, a computer-readable recording medium in which a program for implementing the method performed by the server is recorded may be provided.

According to an embodiment of the present disclosure, an electronic device,

a memory storing one or more instructions;

at least one processor executing the one or more instructions;

The at least one processor by executing the one or more instructions,

Send streaming request and user information to the server,

Receive image data about one or more objects corresponding to the user information from the server,

Rendering is performed based on the received image data,

The image data may include one of the plurality of objects selected based on the type of the electronic device identified based on the user information and the relationship between the information on the user's surrounding environment space and the plurality of objects existing in the virtual environment space. It may be characterized in that it is image data related to the above objects.

According to an embodiment of the present disclosure, users of the electronic device may experience as if they were present in the same space. More specifically, whenever the user's surrounding environment space changes, the server reselects an object to be rendered from among a plurality of objects existing in the virtual environment space, so that users of the electronic device can act as if they exist in the same space. can experience

According to an embodiment of the present disclosure, by integrating the modules for the augmented reality and virtual reality services, the server may provide the extended reality service regardless of the type of the electronic device.

According to an embodiment of the present disclosure, an augmented reality service user may experience a more realistic augmented reality by rendering a virtual object based on information about objects existing in the user's surrounding environment space.

According to an embodiment of the present disclosure, by identifying a relationship between a plurality of objects existing in the virtual environment space and selecting one or more objects from among the plurality of objects based on the identification, the electronic device is a trivial object that has no relationship with other objects. may not be rendered.

1 is a diagram for explaining a schematic network environment according to an embodiment of the present disclosure.
2 is a flowchart illustrating an operation procedure between an electronic device and a server according to an embodiment of the present disclosure.
3 is a flowchart illustrating an operation procedure between electronic devices and a server according to an embodiment of the present disclosure.
4 to 6 are flowcharts illustrating a process in which a server operates based on a type of an electronic device according to an embodiment of the present disclosure.
7A to 7C are diagrams for explaining a process in which a server identifies a relationship between a plurality of objects existing in a virtual environment space according to an embodiment of the present disclosure.
8A to 8F are diagrams illustrating a server according to an embodiment of the present disclosure showing a plurality of objects existing in a virtual environment space based on a type of an electronic device, information about a user's surrounding environment space, and relationships between the plurality of objects; It is a diagram for explaining a process of selecting one or more objects among
9 to 10 are flowcharts illustrating an operation procedure between electronic devices and a server when a user's surrounding environment space is changed according to an embodiment of the present disclosure.
11 is a diagram for schematically explaining an operation procedure between a server and an electronic device.
12A to 12E are diagrams for explaining an example in which an electronic device transmits a streaming request and user information to a server, receives image data from the server, and performs rendering, according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

Hereinafter, in describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present disclosure pertains and are not directly related to the present disclosure will be omitted. This is to more clearly convey the gist of the present disclosure without obscuring the gist of the present disclosure by omitting unnecessary description.

For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings. In addition, the size of each component does not fully reflect the actual size. In each figure, the same or corresponding elements are assigned the same reference numerals.

Advantages and features of the present disclosure, and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the present disclosure to be complete, and those of ordinary skill in the art to which the present disclosure pertains. It is provided to fully inform the person of the scope of the present disclosure, and the present disclosure is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

At this time, it will be understood that each block of the flowchart diagrams and combinations of the flowchart diagrams may be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions performed by the processor of the computer or other programmable data processing equipment are not described in the flowchart block(s). It creates a means to perform functions. These computer program instructions may also be stored in a computer-usable or computer-readable memory that may direct a computer or other programmable data processing equipment to implement a function in a particular manner, and thus the computer-usable or computer-readable memory. It is also possible that the instructions stored in the flow chart block(s) produce an article of manufacture containing instruction means for performing the function described in the flowchart block(s). The computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to create a computer or other programmable data processing equipment. It is also possible that instructions for performing the processing equipment provide steps for performing the functions described in the flowchart block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). It should also be noted that in some alternative implementations it is also possible for the functions recited in blocks to occur out of order. For example, two blocks shown one after another may be performed substantially simultaneously, or the blocks may sometimes be performed in the reverse order according to a corresponding function.

At this time, the term '~ unit' used in this embodiment means software or hardware components such as FPGA or ASIC, and '~ unit' performs certain roles. However, '-part' is not limited to software or hardware. '~' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Accordingly, as an example, '~' indicates components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card. Also, in an embodiment, '~ part' may include one or more processors.

In the present disclosure, a virtual environment is at least one of augmented reality (AR), virtual reality (VR), mixed reality (MR), or extended reality (XR). It refers to a synthetic environment created artificially by adding sensory information including voice along with visual experience to reproduce .

In the present disclosure, augmented reality (AR) refers to overlaying and displaying a virtual image on a physical environment space or real object of the real world.

In the present disclosure, virtual reality (VR) refers to a specific environment or situation that is similar to the real world created by artificial technology but is not real.

In the present disclosure, mixed reality (MR) means showing new information such as a new environment by combining the virtual world and the real world.

In the present disclosure, extended reality (XR) refers to an environment or situation in which the real world and the virtual world are combined to allow humans and machines to interact, or augmented reality, virtual reality or mixed reality technology created by artificial technology. It refers to the concept of freely choosing and creating an expanded reality.

In the present disclosure, an augmented reality device refers to a device capable of providing an augmented reality service, and a virtual reality device refers to a device capable of providing a virtual reality service.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a schematic network environment according to an embodiment of the present disclosure.

Referring to FIG. 1 , the network environment 100 of the present disclosure may include a server 1000 , a plurality of electronic devices 2000 , 2100 , 2200 , and a network 3000 . However, the configuration included in the network environment 100 is not limited thereto. In particular, although three electronic devices 2000 , 2100 , and 2200 are illustrated in FIG. 1 , the network environment 100 may include one, two, or four or more electronic devices.

According to an embodiment, each of the components included in the network environment 100 may mean a physical entity unit or a software or module unit capable of performing an individual function.

According to an embodiment, the server 1000 may be a server that provides content for a user to be immersed in a virtual environment including at least one of augmented reality, virtual reality, mixed reality, and extended reality. For example, the server 1000 may include a server (MEC server), a cloud server, etc. for providing a multi-access edge computing or mobile-edge computing (MEC) service.

According to an embodiment, the electronic devices 2000 , 2100 , and 2200 may refer to devices used by a user. For example, the electronic devices 2000 , 2100 , and 2200 may include a terminal, a user equipment (UE), a mobile station, a subscriber station, a remote terminal, and a wireless terminal. (wireless terminal), or may mean a user device (user device).

In addition, the electronic devices 2000 , 2100 , and 2200 may be terminals that provide content for a user to be immersed in a virtual environment including at least one of augmented reality, virtual reality, mixed reality, and extended reality. .

That is, according to an embodiment, the electronic devices 2000 , 2100 , and 2200 may include a head mounted display (HMD) or a virtual reality headset that provides content for virtual reality, augmented reality, mixed reality, or extended reality. It may be a virtual reality headset (VRH), a pair of glasses-shaped augmented reality glasses worn on the face, or an augmented reality helmet.

According to an embodiment, the network 3000 may provide a channel for wireless communication with the electronic devices 2000 , 2100 , and 2200 . For example, the network 3000 means a radio access network (RAN), a base station, an inode ratio (eNB, eNodeB), a transmission/reception point (TRP, transmission/reception point), or a 5th generation NodeB (5GNB). can do.

According to an embodiment, each of the plurality of electronic devices 2000 , 2100 , and 2200 transmits a streaming request and user information to the server 1000 through the network 3000 , and receives image data from the server 1000 . can A detailed description of an operation between the server 1000 and the electronic devices 2000, 2100, and 2200 based on user information will be described later.

2 is a flowchart illustrating an operation procedure between the electronic device 2000 and the server 1000 according to an embodiment of the present disclosure.

2 illustrates that only one electronic device 2000 of the plurality of electronic devices 2000, 2100, and 2200 performs an operation with the server 1000, each of the plurality of electronic devices 2000, 2100, and 2200 The server 1000 and the above operation may be performed simultaneously (simultaneously) or with a time difference.

Referring to FIG. 2 , in step S210 , the electronic device 2000 may transmit a streaming request and user information to the server 1000 . The streaming request may be a request for information about the virtual environment space. Also, the user information may include information about the environment of the user who uses the electronic device 2000 . For example, the user information may include information about the type of the electronic device 2000 or information about the user's surrounding environment space. Meanwhile, information about the user's surrounding environment space may be acquired by the electronic device 2000 .

In operation S220 , the server 1000 may identify information about the type of the electronic device 2000 and information about the user's surrounding environment space based on the user information. The information about the type of the electronic device 2000 may be information about which service among the augmented reality, virtual reality, mixed reality, or extended reality service can be provided by the electronic device 2000 .

On the other hand, information about the user's surrounding environment space is information about one or more objects existing around the user. ) and information about at least one of rotation of the object. In this case, the one or more objects may include an object corresponding to the background. Meanwhile, information about one or more objects may be expressed as a depth map, a point cloud, a segmentation mask, or a camera matrix, but is not limited thereto.

According to an embodiment, when the server 1000 identifies that the electronic device 2000 is an augmented reality device, the server 1000 may identify information about the user's surrounding environment space. Also, when the server 1000 identifies that the electronic device 2000 is a virtual reality device, the server 1000 may not identify information about the user's surrounding environment space, but is not limited thereto.

Meanwhile, although FIG. 2 illustrates that the server 1000 identifies information about a user's surrounding environment space based on user information, according to an exemplary embodiment, the electronic device 2000 provides a list of objects and types of objects. , the size of the object, the location of the object, or the rotation of the object may identify information about the user's surrounding environment space including information about at least one and transmit the identified information to the server 1000 .

In step S230 , the server 1000 may identify a relationship between a plurality of objects existing in the virtual environment space. A relationship between objects indicates that there is connectivity or dependency between objects, and may be identified based on proximity, semantic reasoning, or movement between objects. In this regard, it will be described in detail with reference to FIGS. 7A to 7C.

In operation S240 , the server 1000 determines one or more objects among the plurality of objects existing in the virtual environment space based on the type of the electronic device 2000 , information about the user's surrounding environment space, and relationships between the plurality of objects. can be selected. In this regard, it will be described in detail with reference to FIGS. 8A to 8F.

In operation S250 , the server 1000 may generate image data regarding one or more selected objects. In this case, the image data may include information about at least one of a list, type, size, position, and rotation of one or more objects selected based on a relationship between the plurality of objects.

In operation S260 , the server 1000 may transmit the generated image data to the electronic device 2000 . In this case, the image data may be expressed as an equirectangular image, a cubic image, and a 3D (3-dimensional) object list and location information, but is not limited thereto.

In operation S270 , the electronic device 2000 may perform rendering based on the received image data.

3 is a flowchart illustrating an operation procedure between the electronic devices 2000 and 2100 and the server 1000 according to an embodiment of the present disclosure. Operations overlapping those of FIG. 2 will be briefly described.

In operation S310 , the first electronic device 2000 may transmit a streaming request and user information to the server 1000 . In operation S315 , the second electronic device 2100 may transmit a streaming request and user information to the server 1000 . The user information may be transmitted separately from the streaming request, but is not limited thereto, and the user information may be included in the streaming request.

In this case, the user information may include information about at least one of a size, a position, and a rotation of each object corresponding to the user who uses the first electronic device 2000 or the second electronic device 2100 .

Although it is illustrated in FIG. 3 that the first electronic device 2000 transmits the streaming request and user information before the second electronic device 2100 , this is only one embodiment, and the second electronic device 2100 may transmit the streaming request and user information before or simultaneously with the first electronic device 2000 . Also, although it is illustrated that the second electronic device 2100 transmits a streaming request and user information before the server 1000 performs operation S320, this is only one embodiment, and the second electronic device 2100 ) may transmit a streaming request and user information at any moment after the server 1000 performs operation S320.

In operation S320 , the server 1000 may identify information about the type of the first or second electronic device 2000 or 2100 and information about the user's surrounding environment space based on the user information.

In operation S325 , the server 1000 may determine (or generate) information about the virtual environment space based on media content or information about the user's surrounding environment space. The information about the virtual environment space may include information about at least one of a list of objects existing in the virtual environment space, a type of an object, a size of an object, a position of an object, and rotation of an object.

When the server 1000 determines information about the virtual environment space based on the media content, the server 1000 identifies one or more objects existing in the media content, and maps the information about the identified one or more objects to a depth map, It may be expressed as a point cloud, a segmentation mask, or a camera matrix, but is not limited thereto.

According to an embodiment, the media content is previously generated and input to the server 1000 or existing in the server 1000 , and may include a moving picture, live-stream content, or a virtual reality image. In this case, the virtual reality image may mean an equirectangular projection image or a cubic projection image, but is not limited thereto.

When the server 1000 determines the information about the virtual environment space based on the information about the user's surrounding environment space, according to an embodiment, the server 1000 is the first or second electronic device 2000 or 2100 . It is possible to select one piece of user information from among the user information received from , and determine information about the virtual environment space based on the selection. Also, according to an embodiment, the server 1000 may determine information about two or more virtual environment spaces based on each of the user information received from the first or second electronic devices 2000 and 2100 .

In operation S330 , the server 1000 may identify a relationship between a plurality of objects existing in the virtual environment space.

In operation S340 , the server 1000 selects one or more objects from among the plurality of objects based on the type of the first or second electronic device 2000 or 2100 , the user's surrounding environment space, and a relationship between the plurality of objects. can

In operation S350 , the server 1000 may generate image data regarding one or more selected objects. In this case, the image data may include information about an object corresponding to the user of the first or second electronic device 2000 or 2100 . According to an embodiment, image data to be transmitted to the first electronic device 2000 may include information about an object corresponding to a user of the second electronic device 2100 , and is transmitted to the second electronic device 2100 . The to-be-formed image data may include information about an object corresponding to the user of the first electronic device 2000 .

In operation S360 , the server 1000 may transmit the generated image data to the first or second electronic device 2000 or 2100 . In operation S370 , the first or second electronic device 2000 or 2100 may perform rendering based on the received image data.

4 to 6 are flowcharts illustrating a process in which the server 1000 operates based on the types of the electronic devices 2000 , 2100 , and 2200 according to an embodiment of the present disclosure. Operations overlapping those of FIGS. 2 and 3 will be briefly described.

Referring to FIG. 4 , in step S410 , the server 1000 may receive a streaming request and user information from the electronic devices 2000 , 2100 , and 2200 . In operation S420 , the server 1000 may identify information about the type of the electronic devices 2000 , 2100 , and 2200 and information about the user's surrounding environment space based on the user information. In operation S430 , the server 1000 may determine information about the first virtual environment space based on media content or information about the user's surrounding environment space. In operation S440 , the server 1000 may identify a relationship between a plurality of objects existing in the first virtual environment space.

In operation S450 , the server 1000 may identify the types of the electronic devices 2000 , 2100 , and 2200 .

5A is a flowchart illustrating a method in which the server 1000 selects one or more objects from among a plurality of objects when the electronic devices 2000 , 2100 , and 2200 are augmented reality devices as a result of identification in step S450 .

In operation S510 , the server 1000 may select one or more objects from among the plurality of objects based on at least one object existing in the user's environment space and a relationship between the plurality of objects. Information on at least one object existing in the user's surrounding environment space may be obtained based on user information received from the electronic devices 2000 , 2100 , and 2200 .

5B is a flowchart illustrating a method in which the server 1000 selects one or more objects from among a plurality of objects when the electronic devices 2000 , 2100 , and 2200 are virtual reality devices as a result of identification in step S450 .

In operation S520 , the server 1000 may identify whether the electronic devices 2000 , 2100 , and 2200 have rendered the second virtual environment space based on default image data. According to an embodiment, the default image data is stored in the electronic devices 2000 and 2100 before the electronic devices 2000 , 2100 , and 2200 transmit a streaming request to the server 1000 or before receiving the image data from the server 1000 . , 2200) may mean image data existing in the . Also, according to an embodiment, no object may exist in the virtual environment space created based on the default image data.

In operation S530 , when the identification result in operation S520 is No, the server 1000 may identify whether the second virtual environment space is the same as the first virtual environment space.

According to an embodiment, that the identification result in step S520 is No means that the electronic devices 2000 , 2100 , and 2200 have rendered the second virtual environment space based on image data previously received from the server 1000 . can do. Also, the image data previously received from the server 1000 may mean the most recent image data among the image data received by the electronic devices 2000 , 2100 , and 2200 from the server 1000 .

In operation S540 , when the identification result in operation S530 is Yes, the server 1000 may select an object corresponding to a user of another electronic device. According to an embodiment, that the identification result in step S530 is Yes may mean that the server 1000 has determined information about the first virtual environment space based on the information about the second virtual environment space.

In operation S550 , when the identification result in operation S530 is No, the server 1000 may determine whether to render one or more objects among a plurality of objects existing in the first virtual environment space in the second virtual environment space. Rendering one or more objects among the plurality of objects existing in the first virtual environment space in the second virtual environment space means that one or more objects to be selected in a subsequent process are rendered and added to the second virtual environment space rendered by the electronic device. can mean that

According to an embodiment, the server 1000 may receive a user input and determine whether to render one or more objects among a plurality of objects existing in the first virtual environment space in the second virtual environment space based on the received user input. According to another embodiment, the server 1000 may determine whether to render one or more objects among a plurality of objects existing in the first virtual environment space in the second virtual environment space based on a predetermined setting.

In operation S560, when the identification result in operation S520 is Yes or the determination result in operation S550 is No, the server 1000 may select all objects existing in the first virtual environment space. In this case, all objects existing in the first virtual environment space may include objects corresponding to the background of the first virtual environment space.

In step S570 , when the determination result in step S550 is Yes, the server 1000 may identify information about the second virtual environment space. The information about the second virtual environment space may include information about at least one of a list of objects existing in the second virtual environment space, a type of an object, a size of an object, a position of an object, and a rotation of the object.

In operation S580 , the server 1000 may select one or more objects from among the plurality of objects based on at least one object existing in the second virtual environment space and a relationship between the plurality of objects.

Referring to FIG. 6 , in step S610 , the server 1000 may generate image data regarding one or more selected objects. In operation S620 , the server 1000 may transmit the generated image data to the electronic devices 2000 , 2100 , and 2200 .

7A to 7C are diagrams for explaining a process in which the server 1000 identifies a relationship between a plurality of objects existing in a virtual environment space according to an embodiment of the present disclosure.

7A illustrates a process of identifying a relationship based on a degree of proximity between a plurality of objects existing in a virtual environment space, according to an embodiment of the present disclosure.

Referring to FIG. 7A , a segmentation mask 510 is illustrated as a result of identifying a plurality of objects existing in the virtual environment space. According to an embodiment, a person 712 , a drinking cup 714 , a background 716 , and a table 718 present in the virtual environment space may be identified. Although the segmentation mask 710 is illustrated as a result of identifying a plurality of objects existing in the virtual environment space, the present invention is not limited thereto.

According to an embodiment, the server 1000 may identify a relationship based on a contact area between a plurality of objects or a distance between centers. For example, when the area in contact with the second object among the total area of the first object is equal to or greater than a preset value, the server 1000 may determine that a relationship exists between the first object and the second object. Alternatively, when the distance between the center of the first object and the center of the second object is equal to or less than a preset value, the server 1000 may determine that a relationship between the first object and the second object exists.

For example, the server 1000 may determine that there is a relationship between the drink cup 714 and the person 712 because the area in contact with the person 712 among the total area of the drink cup 714 is equal to or greater than a preset value. . Also, since the distance between the center of the person 712 and the center of the table 718 is equal to or less than a preset value, the server 1000 may determine that a relationship exists between the person 712 and the table 718 .

When the server 1000 identifies the relationship between the objects, the server 1000 may identify the positional relationship between the objects in which the relationship exists. According to an embodiment, the server 1000 may identify a positional relationship between objects in which a relationship exists, and represent this as a vector. For example, the server 1000 may represent a position difference between the centers of the cup 714 and the person 712 as a vector 722 . In addition, the server 1000 may represent the position difference between the center of the person 712 and the table 718 as vectors 724 and 726 . A vector is a physical quantity having a direction, and may appear in one direction or in both directions depending on an object that is a criterion for judgment. For example, when the position difference with the person 712 with respect to the cup 714 is expressed as a vector, the point of action of the vector 722 is located on the cup 714 and the direction of the vector 722 is the person 712 . can be headed Meanwhile, although a vector is illustrated as a method of indicating a positional relationship between objects, this is only an example and is not limited thereto.

7B illustrates a process of identifying a relationship based on semantic reasoning between a plurality of objects existing in a virtual environment space, according to an embodiment of the present disclosure.

Referring to FIG. 7B , a result 730 of identifying a plurality of objects existing in a virtual environment space using the You Only Look One (YOLO) algorithm is shown. According to an embodiment, the boy 732 , the fork 734 , the children 736 , the table 738 , and the dessert 740 present in the virtual environment space may be identified. A result of identifying a plurality of objects existing in the virtual environment space using the YOLO algorithm is illustrated, but is not limited thereto. Meanwhile, the segmentation mask 710 or the identification result 730 using the YOLO algorithm may be obtained in the process of identifying one or more objects existing in the user's surrounding environment space or media content.

According to an embodiment, when the server 1000 can infer an action or a positional relationship that can be performed between objects based on the meaning of the first object and the meaning of the second object, the first object and the second object It can be determined that a relationship between objects exists.

Referring to FIG. 7B , for example, since server 1000 can infer that boy 732 can hold fork 734 , server 1000 determines that the relationship between boy 732 and fork 734 is can be considered to exist. Also, since server 1000 may infer that children 736 may sit around table 738 , server 1000 may determine that a relationship between children 736 and table 738 exists. Since the server 1000 may infer that the children 736 can eat the dessert 740 , the server 1000 may determine that a relationship between the children 736 and the dessert 740 exists.

Meanwhile, although FIGS. 7A to 7B show an embodiment in which the server 1000 identifies a relationship based on proximity or semantic inference between a plurality of objects, this is only an embodiment, and the server 1000 A relationship between the plurality of objects may be identified based on other factors. For example, the server 1000 may identify the relationship based on the movement of the object. More specifically, the server 1000 may determine that the relationship exists based on the movement of the object from different starting points in the same direction and speed or from different starting points to the same place.

According to an embodiment, the server 1000 may identify a relationship between objects based on one factor described above, but is not limited thereto, and may identify a relationship between objects based on two or more factors.

7C illustrates a table 750 corresponding to a result of the server 1000 identifying a relationship between a plurality of objects existing in a virtual environment space, according to an embodiment of the present disclosure.

The object column 752 may include information about a plurality of objects existing in the virtual environment space.

The time column 754 may include information regarding a time at which a relationship between objects was identified. Meanwhile, although the table 750 shows the time at which the relationship between objects is identified divided based on seconds, it may be divided based on the frame number, but is not limited thereto.

The location column 756 and the size column 758 may include information regarding the location and size of the identified object. According to an embodiment, the position of the identified object may be expressed as the position of the center of the object, and the size of the object may be expressed as the area of the object, but is not limited thereto.

The relationship column 760 may include information about an object determined to have a relationship with the object of the object column 752 at a specific time. Although table 750 shows that only one object has a relationship with an object in object column 752 at a particular time, according to one embodiment, two or more objects at a particular time are objects in object column 752 . It can be judged that a relationship exists with

The vector column 762 may include information about a position difference between centers of objects for which a relationship is determined to exist.

Meanwhile, although the items of the table 750 include a location column 756 and a size column 758, etc., this is only an example, and one or more objects existing in the user's surrounding environment space or media content are identified. The items of the table 750 may vary according to a method of expressing the result.

8A to 8F are diagrams illustrating a server 1000 in a virtual environment space based on a type of an electronic device, information about a user's surrounding environment space, and relationships between a plurality of objects, according to an embodiment of the present disclosure. It is a diagram for explaining a process of selecting one or more objects from among the objects of . The content overlapping with FIGS. 7A to 7C will be briefly described.

8A illustrates an environment space around a user of an augmented reality device, according to an embodiment of the present disclosure. A table 802 , a chessboard 804 , a sofa 806 , and a cushion 808 may exist in the environment space around the user.

8B illustrates a table 820 corresponding to a result of the server 1000 identifying one or more objects existing in the user's surrounding environment space, according to an embodiment of the present disclosure. The object column 822 may include information about one or more objects 802 to 808 existing in the environment space around the user. The location column 824 and the size column 826 may include information regarding the location and size of the identified object.

FIG. 8C illustrates a table 840 corresponding to a result of the server 1000 identifying a relationship between a plurality of objects existing in a virtual environment space, according to an embodiment of the present disclosure. The table 840 of FIG. 8C may indicate the same identification result as the table 750 of FIG. 7C .

FIG. 8D illustrates a table 860 corresponding to a result of the server 1000 selecting one or more objects based on a relationship between a plurality of objects existing in a virtual environment space, according to an embodiment of the present disclosure. Table 860 may further include a time column 862 , an associated object column 864 , and a vector column 866 compared to the table 820 of FIG. 8B .

According to an embodiment, the server 1000 may identify an object corresponding to an object existing in the user's surrounding environment space from among a plurality of objects existing in the virtual environment space. For example, the server 1000 may identify the table 846 corresponding to the table 802 existing in the user's surrounding environment space from among the plurality of objects existing in the virtual environment space. Next, the server 1000 may select an object related to the identified object based on the table 840 and input it into the related object column 864 . For example, based on the table 840 , the server 1000 may select a person 842 with a relationship to the table 846 and input it into the related object column 864 . According to an embodiment, the server 1000 obtains information about a vector indicating a time when a relationship is identified and a position difference between objects with a relationship, based on the table 840 , and uses this information as a time column of the table 860 . 862 and vector column 866, respectively. For example, based on the table 840, the server 1000 may provide information about the time at which the relationship between the table 846 and the person 842 was identified 0:03-0:04 and 0:04-0: 05 may be obtained, and vectors {10, 256} and {10, 258} indicating the position difference between the table 846 and the person 842 at the corresponding time may be obtained. Thereafter, the server 1000 may input the obtained time and vector information into the time column 862 and the vector column 866 of the table 860 , respectively.

8E is a table corresponding to a result of the server 1000 additionally selecting an object related to the selected object based on the relationship between a plurality of objects existing in the virtual environment space, according to an embodiment of the present disclosure; 880 is shown. Operations overlapping with the above-described operations will be briefly described.

According to an embodiment, the server 1000 may additionally input the object input in the related object column 864 into the object column 822 . For example, the server 1000 may additionally input the person input in the related object column 864 into the object column 822 . Next, the server 1000 may determine the size and position of the object additionally input to the object column 822 based on the table 840 . For example, the server 1000 determines the position and size of the person 882 based on a vector representing the difference between the positions of the table 846 and the person 842 , and determines the position and size of the person 882 based on the location column 824 and the object column 826 . ) can be entered. In this case, the server 1000 may adjust the location and size of the person 882 based on information about the user's surrounding environment space.

According to an embodiment, the server 1000 may repeatedly perform the operation described with reference to FIG. 8D for the newly added person 882 . For example, the server 1000 may identify the person 842 corresponding to the person 882 , and input information about the drinking cup 844 related to the person into the table 880 . In this case, the server 1000 may input a vector in a direction opposite to the vector of the table 840 in consideration of the person 882 as the object as a criterion for determination.

Thereafter, the server 1000 may repeatedly perform the above operations until it cannot identify an object related to the additionally input object, or may repeatedly perform the above operations for a preset number of times.

8F illustrates a result of rendering image data regarding a selected object from among a plurality of objects existing in a virtual environment space, according to an embodiment of the present disclosure. For example, the electronic devices 2000 , 2100 , and 2200 may receive image data including the selected person 882 and a drink cup 884 and render the image data.

Meanwhile, although FIGS. 8A to 8F illustrate a method in which the server 1000 operates when the electronic devices 2000 , 2100 , and 2200 are augmented reality devices, the server 1000 includes the electronic devices 2000 , 2100 , and 2200 . The above operations may be performed even when is a virtual reality device.

9 to 10 are flowcharts illustrating an operation procedure between the electronic devices 2000 and 2100 and the server 1000 when a user's surrounding environment space is changed according to an embodiment of the present disclosure. Operations overlapping those of FIGS. 2 and 3 will be briefly described.

9 is a diagram illustrating a case in which the server 1000 does not determine information about a virtual environment space based on information about a user's surrounding environment space of the first electronic device 2000 according to an embodiment of the present disclosure; An operation procedure between the device 2000 and the server 1000 is shown.

In operation S910 , the first electronic device 2000 may transmit second user information to the server 1000 . According to an embodiment, the second user information may include information about the user's changed surrounding environment space. In this case, that the user's surrounding environment space has changed may mean that information about at least one of a list, type, size, position, or rotation of an object existing in the user's surrounding environment space has changed.

In operation S920 , the server 1000 may identify information about the user's changed surrounding environment space based on the second user information.

In step S930 , the server 1000 may select one or more objects from among the plurality of objects based on the type of the electronic device, information about the user's changed surrounding environment space, and relationships between the plurality of objects.

In operation S940 , the server 1000 may generate image data regarding one or more selected objects. According to an embodiment, information about at least one of a list, type, size, position, and rotation of one or more objects included in the image data may be the same as or different from information before the user's surrounding environment space is changed.

In operation S950 , the server 1000 may transmit the generated image data to the first electronic device 2000 . In operation S960, the first electronic device 2000 may perform rendering based on the received image data.

10 is a diagram illustrating first and second cases when the server 1000 determines information about a virtual environment space based on information about a user's surrounding environment space of the first electronic device 2000 according to an embodiment of the present disclosure; An operation procedure between the electronic devices 2000 and 2100 and the server 1000 is illustrated.

In step S1010 , the first electronic device 2000 may transmit second user information to the server 1000 . According to an embodiment, the second user information may include information about the user's changed surrounding environment space.

In step S1020 , the server 1000 may identify information about the user's changed surrounding environment space based on the second user information.

In step S1025 , the server 1000 may determine information about the second virtual environment space based on the user's changed information about the surrounding environment space.

In operation S1030 , the server 1000 may identify a relationship between a plurality of second objects existing in the second virtual environment space. The relationship between the second plurality of objects may be the same as or different from the relationship between the plurality of objects before the user's surrounding environment space is changed.

In operation S1040 , the server 1000 may select one or more objects from among the plurality of objects based on the type of the electronic device, the information about the user's surrounding environment and space, and the relationship between the second plurality of objects.

In operation S1050 , the server 1000 may generate image data regarding one or more selected objects.

In operation S1060 , the server 1000 may transmit the generated image data to the first or second electronic device 2000 or 2100 . In operation S1070 , the first or second electronic device 2000 or 2100 may perform rendering based on the received image data. According to an embodiment, the server 1000 may not transmit the generated image data to the first electronic device 2000 that has transmitted the second user information.

11 is a diagram schematically illustrating an operation procedure between the server 1000 and the electronic device 2000 .

According to an embodiment, the server 1000 may include a network interface 1110 , a processor 1120 , and a memory 1130 . However, the configuration of the server 1000 is not limited to the above, and may include more or fewer configurations.

The server 1000 receives a streaming request and user information 1142 from the electronic device 2000 using the network interface 1110 , and transmits image data 1144 regarding one or more selected objects to the electronic device 2000 . can

According to an embodiment, the processor 1120 may control the overall operation of the server 1000 by executing one or more instructions in the memory 1130 .

For example, the processor 1120 may identify one or more objects existing in the environment space around the user based on user information received from the electronic device 2000 . Also, the processor 1120 may identify one or more objects existing in the media content.

The processor 1120 may determine information about the virtual environment space based on media content or information about the user's surrounding environment space. Also, the processor 1120 may identify a relationship between a plurality of objects existing in the virtual environment space.

The processor 1120 may select one or more objects from among the plurality of objects based on the type of the electronic device 2000 , the user's surrounding environment space, and a relationship between the plurality of objects. Also, the processor 1120 may generate image data regarding one or more selected objects.

The processor 1120 may transmit the generated image data 1144 to the electronic device 2000 .

With respect to the operations of the processor 1120, since it has been described above with reference to FIGS. 2 and 3, the detailed description will be omitted.

According to an embodiment, the memory 1130 may include an object identification module 1132 that stores instructions for the server 1000 to identify one or more objects existing in the user's surrounding environment space or media content.

The memory 1130 may include a virtual environment space management module 1134 that stores instructions for the server 1000 to determine information about the virtual environment space based on media content or information about the user's surrounding environment space. have. In this case, the virtual environment space management module 1134 is a module for determining information about the virtual environment space and managing information about objects existing in the virtual environment space, and may mean a 3D game engine or a 3D simulation module. .

The memory 1130 may include a relationship identification module 1136 in which the server 1000 stores instructions for identifying a relationship between a plurality of objects existing in the virtual environment space. Also, the memory 1130 allows the server 1000 to select one or more objects from among the plurality of objects based on the type of the electronic device 2000, information about the user's surrounding environment space, and relationships between the plurality of objects. and an object selection module 1138 for storing instructions for

The memory 1130 may include a scene composition module 1140 that stores instructions for generating image data related to one or more objects selected by the server 1000, but is not limited thereto.

According to an embodiment, the electronic device 2000 may include a network interface 1150 , a sensing module 1160 , a processor 1170 , and a memory 1180 . However, the configuration of the electronic device 2000 is not limited to the above, and may include more or fewer configurations.

The electronic device 2000 may transmit a streaming request and user information 1142 to the server 1000 using the network interface 1150 , and may receive image data 1144 related to one or more selected objects from the server 1000 . have.

The electronic device 2000 may use the sensing module 1160 to obtain information about the user's surrounding environment space. Also, the electronic device 2000 may use the sensing module 1160 to obtain a user input regarding whether to render one or more objects among a plurality of objects existing in the first virtual environment space in the second virtual environment space. can

According to an embodiment, the processor 1170 controls the overall operation of the electronic device 2000 by executing one or more instructions in the memory 1180 . For example, the processor 1170 may control the sensing module 1160 , the network interface 1150 , and the like by executing one or more instructions stored in the memory 1180 .

According to an embodiment, the memory 1180 may include a rendering module 1182 that stores instructions for performing rendering based on the image data 1144 received from the server 1000, but is limited thereto. not.

12A to 12E show that the electronic devices 2000 , 2100 , and 2200 transmit a streaming request and user information to the server 1000 , and receive image data from the server 1000 to render them, according to an embodiment of the present disclosure. It is a diagram for explaining an example of performing the

Referring to FIG. 12A , according to an embodiment, the first and second electronic devices 2000 and 2100 may be augmented reality devices, and the third electronic device 2200 may be virtual reality electronic devices. According to an embodiment, only the table 1202 may exist in the environment space (empty office) of the user of the first electronic device 2000 . In addition, a table 1212 , a laptop computer 1214 , a cushion 1216 , and a chair 1218 may exist in an environment space (a conference room) around the user of the second electronic device 2100 . The third electronic device 2200 may determine information about the second virtual environment space based on the default image data. This may mean that no object exists in the second virtual environment space rendered by the third electronic device. However, this is only an example, and the third electronic device 2200 may render a second virtual environment space in which one or more objects exist based on previously received image data.

According to an embodiment, the electronic devices 2000 , 2100 , and 2200 may transmit a streaming request and first user information to the server 1000 . The first user information may include information about the type of the electronic device 2000 , 2100 , or 2200 or information about the user's surrounding environment space. For example, the first user information transmitted by the first electronic device 2000 may include information indicating that the first electronic device 2000 is an augmented reality device and information about a user's surrounding environment space (office). have. Also, the first user information transmitted by the second electronic device 2100 may include information indicating that the second electronic device 2100 is an augmented reality device and information about a user's surrounding environment space (conference room). The first user information transmitted by the third electronic device 2200 is information indicating that the third electronic device 2200 is a virtual reality device and the third electronic device 2200 creates a second virtual environment space based on default image data. It may include information on whether or not it has been rendered. However, information included in the first user information is not limited thereto.

According to an embodiment, the server 1000 may determine information about the first virtual environment space based on media content or information about the user's surrounding environment space. For example, the server 1000 may determine information about the first virtual environment space based on information about the environment space (conference room) around the user of the second electronic device 2100 . Alternatively, the server 1000 may determine information about the first virtual environment space based on information about the second virtual environment space rendered by the third electronic device 2200 .

According to an embodiment, the server 1000 may identify a relationship between a plurality of objects existing in the first virtual environment space. For example, the server 1000 may identify a relationship between objects existing in the first virtual environment space. More specifically, the server 1000 may determine that the relationship between the table 1212 and the notebook 1214 and the relationship between the cushion 1216 and the chair 1218 exist.

According to an embodiment, the server 1000 may configure a plurality of devices existing in the virtual environment space based on the type of the electronic device 2000 , 2100 , and 2200 , information about the user's surrounding environment space, and relationships between the plurality of objects. One or more of the objects may be selected. For example, the server 1000 may configure the second and third electronic devices 2100 and 2200 based on the type of the first electronic device 2000 , information about the user's surrounding environment space, and relationships between a plurality of objects. You can select an object and notebook 1214 corresponding to the user of . In addition, the server 1000 determines the user of the first and third electronic devices 2000 and 2200 based on the type of the second electronic device 2100 , information about the user's surrounding environment space, and relationships between a plurality of objects. You can select the object corresponding to . The server 1000 may select all objects including objects corresponding to users of the first and second electronic devices 2000 and 2100 based on the type of the third electronic device 2200 and the relationship between the plurality of objects. have.

According to an embodiment, the server 1000 may generate image data regarding one or more selected objects, and transmit the generated image data to the electronic devices 2000 , 2100 , and 2200 .

Referring to FIG. 12B , the electronic devices 2000 , 2100 , and 2200 may perform rendering based on the received image data. For example, the first electronic device 2000 may render the objects 1280 and 1290 and the notebook 1204 corresponding to users of the second and third electronic devices 2100 and 2200 . Also, the second electronic device 2100 may render objects 1270 and 1290 corresponding to users of the first and third electronic devices 2000 and 2200 . The third electronic device 2200 includes objects 1270 and 1280 corresponding to users of the first and second electronic devices 2000 and 2100 , a table 1222 , a notebook 1224 , a cushion 1226 , and a chair 1228 . ) can be rendered. Accordingly, users of the electronic devices 2000 , 2100 , and 2200 may experience as if they exist in the same space. In this case, the sizes and positions of the rendered objects may be changed based on information about the user's surrounding environment space.

Meanwhile, FIGS. 12B to 12E show the first and second electronic devices 2000 and 2100 in order to distinguish objects existing in the user's surrounding environment from objects rendered by the first and second electronic devices 2000 and 2100. ) shows the outline of the rendered object in bold.

According to an embodiment, when the user's surrounding environment space of the electronic device 2000 , 2100 , or 2200 is changed, the electronic device 2000 , 2100 , 2200 includes information about the changed surrounding environment space to the server 1000 . to transmit second user information.

Referring to FIG. 12C , for example, when a chair 1206 is newly added to the environment space of the user of the first electronic device 2000 , the first electronic device 2000 displays information about the chair 1206 . The additionally included second user information may be transmitted. Next, on the basis of the changed information on the surrounding environment space and the relationship between the plurality of objects, the server 1000 sets the object corresponding to the user of the second and third electronic devices 2100 and 2200, the notebook 1214 and the The cushion 1216 may be selected and image data regarding the selected objects may be transmitted to the first electronic device 2000 . The first electronic device may render the objects 1280 and 1290, the notebook 1204, and the cushion 1208 corresponding to the users of the second and third electronic devices 2100 and 2200 based on the received data.

Referring to FIG. 12D , for example, when the notebook 1214 is removed from the environment space around the user of the second electronic device 2100, the second electronic device 2100 does not include information about the notebook 1214. It is possible to transmit second user information that is not Next, the server 1000 selects an object corresponding to the user of the second and third electronic devices 2100 and 2200 based on the changed information on the surrounding environment space and the relationship between the plurality of objects, and selects the selected object may transmit image data regarding to the first electronic device 2000 . In addition, the server 1000 selects an object, a table 1212 , a cushion 1216 , and a chair 1218 corresponding to users of the first and second electronic devices 2000 and 2100 , and image data related to the selected object may be transmitted to the third electronic device 2200 .

According to an embodiment, the first electronic device 2000 may render objects 1280 and 1290 corresponding to users of the second and third electronic devices 2100 and 2200 based on the received data. Also, the third electronic device 2200 renders the objects 1270 and 1280 , the table 1222 , the cushion 1226 , and the chair 1228 corresponding to the users of the first and second electronic devices 2000 and 2100 . can do.

Referring to FIG. 12E , when the position and rotation of the notebook 1214 in the environment space around the user of the second electronic device 2100 are changed, the second electronic device 2100 includes information about the changed notebook 1244 . to transmit second user information. Next, the server 1000 may transmit the changed image data regarding the notebook 1244 to the first and third electronic devices 2000 and 2200 .

According to an embodiment, the first electronic device 2000 may render the notebook 1234 whose position and rotation are changed based on the changed image data about the notebook 1244 . Also, the third electronic device 2200 may render the notebook 1254 whose position and rotation have been changed based on the changed image data about the notebook 1244 .

When the server 1000 performs the above-described operation whenever the user's surrounding environment space changes, users of the electronic devices 2000 , 2100 , and 2200 can experience as if they exist in the same space.

Claims (20)

a memory storing one or more instructions; and
at least one processor executing the one or more instructions;
The at least one processor by executing the one or more instructions,
Receive a streaming request and first user information from the electronic device,
based on the first user information, identify information about a type of the electronic device and a user's surrounding environment space,
Identifies a relationship between a plurality of objects existing in the first virtual environment space,
selecting one or more objects from among the plurality of objects based on the type of the electronic device, information about the user's surrounding environment space, and a relationship between the plurality of objects,
A server that transmits image data regarding the selected one or more objects to the electronic device According to claim 1,
The at least one processor is configured to determine the information about the first virtual environment space based on media content or information about the user's surrounding environment space. According to claim 1,
When selecting one or more objects from among the plurality of objects, if the electronic device is an augmented reality device, the at least one processor is configured to control at least one object existing in the environment space surrounding the user and a relationship between the plurality of objects Characterized in selecting based on the, server. According to claim 1,
When one or more objects are selected from among the plurality of objects, if the electronic device is a virtual reality device, the at least one processor is configured to provide information on a second virtual environment space rendered by the electronic device and information about the space between the plurality of objects. A server, characterized in that the selection is based on a relationship. According to claim 1,
When the at least one processor identifies a relationship between the plurality of objects existing in the first virtual environment space, the plurality of objects are selected based on at least one of a contact area between the plurality of objects and a center-to-center distance. A server, characterized in that it identifies a relationship between them. According to claim 1,
When the at least one processor identifies the relationship between the plurality of objects existing in the first virtual environment space, the at least one processor identifies the relationship between the plurality of objects based on semantic reasoning. that the server. The server of claim 1 , wherein the information on the relationship between the plurality of objects includes information on the position relationship between the plurality of objects. According to claim 1,
When the at least one processor selects one or more objects from among the plurality of objects, at least one object existing in the environment space surrounding the user or at least one object present in a second virtual environment space rendered by the electronic device Select the first object among the objects,
Based on the relationship between the plurality of objects, the server, characterized in that selecting a second object having a relationship with the first object. 9. The method of claim 8,
The server, characterized in that the at least one processor selects a third object having a relationship with the second object based on a relationship between the plurality of objects. According to claim 1,
The at least one processor receives second user information from the electronic device,
The second user information is characterized in that it includes information about the changed surrounding environment space of the user, the server. A method performed by a server comprising:
Receiving a streaming request and first user information from the electronic device;
identifying information about a type of the electronic device and a user's surrounding environment space based on the first user information;
identifying a relationship between a plurality of objects existing in a first virtual environment space;
selecting one or more objects from among the plurality of objects based on the type of the electronic device, information about the user's surrounding environment space, and relationships between the plurality of objects; and
and transmitting image data regarding the selected one or more objects to the electronic device. 12. The method of claim 11,
The method further comprising the step of determining information about the first virtual environment space based on media content or information about the surrounding environment space of the user. The method of claim 11, wherein selecting one or more objects from among the plurality of objects comprises:
When the electronic device is an augmented reality device, the method comprising the step of selecting based on at least one object existing in a space surrounding the user and a relationship between the plurality of objects. The method of claim 11, wherein selecting one or more objects from among the plurality of objects comprises:
When the electronic device is a virtual reality device, the method comprising: selecting based on information about a second virtual environment space rendered by the electronic device and a relationship between the plurality of objects. The method of claim 11 , wherein the step of identifying a relationship between a plurality of objects existing in the first virtual environment space comprises:
identifying a relationship between the plurality of objects based on at least one of a center-to-center distance and a contact area between the plurality of objects; The method of claim 11 , wherein the step of identifying a relationship between a plurality of objects existing in the first virtual environment space comprises:
and identifying a relationship between the plurality of objects based on semantic inference. The method of claim 11, wherein selecting one or more objects from among the plurality of objects comprises:
selecting a first object from among at least one object existing in an environment space surrounding the user or at least one object existing in a second virtual environment space rendered by the electronic device; and
and selecting a second object having a relationship with the first object based on the relationship between the plurality of objects. The method of claim 17, wherein selecting one or more objects from among the plurality of objects comprises:
based on the relationship between the plurality of objects, further comprising selecting a third object having a relationship with the second object. 12. The method of claim 11,
Further comprising the step of receiving second user information from the electronic device,
The second user information includes information about the changed surrounding environment space of the user. A computer-readable recording medium in which a program for executing the method of any one of claims 11 to 19 in a computer is recorded.

Images