KR20230106257A - System, electronic device and method for controlling external electronic device in real world in virtual enviroment - Google Patents

Abstract

According to various embodiments, a system capable of controlling an external electronic device in the real world in a virtual environment, in a first mode capable of controlling an external electronic device in a virtual environment, based on appearance information and function information of the external electronic device A virtual external electronic device corresponding to the external electronic device is displayed, and when the occurrence of an input event is detected in the virtual external electronic device, input information related to the input event is transmitted to the external electronic device, and from the external electronic device When output information related to the state of the external electronic device changed according to the execution of the function corresponding to the input information is received, the state of the virtual external electronic device is made to correspond to the changed state of the external electronic device based on the output information. electronic devices that change; and when the input information is received from the electronic device, a function corresponding to the input information is performed, and output information related to a state of the external electronic device changed according to the execution of the function corresponding to the input information is transmitted to the electronic device. It may include an external electronic device that does. Other various embodiments are possible.

Description

System, electronic device and method for controlling external electronic devices in the real world in a virtual environment

Various embodiments relate to a system, electronic device, and method capable of controlling an external electronic device in the real world in a virtual environment.

Virtual Reality (VR) is an artificial environment that presents a virtual world created by a computer system to users, which is similar to reality but is not real. and spatial experience.

Augmented Reality (AR) is a field derived from virtual reality, which synthesizes virtual objects (objects, information, etc.) in the real world to make them appear as if they exist in the real world. Based on the combination with a virtual object that augments information, the effect of reality can be further increased.

In the virtual environment composed of the virtual reality and the augmented reality, the user experience required in the virtual environment is diversified and diversified by performing activities directly connected to reality in various aspects (eg, law, finance, entertainment, learning, etc.) and various technologies have been developed for a more realistic experience.

The experience experienced by users in a virtual environment is separated from reality. Objects interacting in virtual reality (VR) are all objects reconstructed in a virtual environment, and interactions with the objects do not affect reality. For example, the act of picking up an apple in Virtual Reality (VR) has no effect on the apple in reality. In addition, when objects in the real world are projected in Augmented Reality (AR), it is difficult to completely interact with them as they are separated from the virtual environment.

As described above, the interaction between the virtual environment and the real world makes it difficult for users to immerse themselves in the virtual environment, and makes users feel that the virtual environment is only an imitation of the real world and does not completely fuse with the real world. It can be an obstacle to the goal of reconstructing reality.

The technical problem to be achieved in the present disclosure is not limited to the technical problem mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

According to various embodiments, a system capable of controlling an external electronic device in the real world in a virtual environment, in a first mode capable of controlling an external electronic device in a virtual environment, based on appearance information and function information of the external electronic device A virtual external electronic device corresponding to the external electronic device is displayed, and when the occurrence of an input event is detected in the virtual external electronic device, input information related to the input event is transmitted to the external electronic device, and from the external electronic device When output information related to the state of the external electronic device changed according to the execution of the function corresponding to the input information is received, the state of the virtual external electronic device is made to correspond to the changed state of the external electronic device based on the output information. electronic devices that change; and when the input information is received from the electronic device, a function corresponding to the input information is performed, and output information related to a state of the external electronic device changed according to the execution of the function corresponding to the input information is transmitted to the electronic device. It may include an external electronic device that does.

According to various embodiments, an electronic device capable of controlling an external electronic device in a real world in a virtual environment includes a camera module and a processor, and the processor performs a first mode capable of controlling an external electronic device in a virtual environment. displays a virtual external electronic device based on appearance information and function information of the external electronic device, and transmits input information related to the input event to the external electronic device when the occurrence of an input event is detected in the virtual external electronic device; , When receiving output information related to the state of the external electronic device changed according to the execution of the function corresponding to the input information from the external electronic device, the state of the virtual external electronic device is determined based on the output information. display to correspond to the state of the device, and when output information related to an event generated in the external electronic device is received from the external electronic device, the virtual external electronic device corresponds to the state of the external electronic device changed based on the output information Can be set to change the state of the device.

A method for controlling an external electronic device in the real world in a virtual environment in a system according to various embodiments includes, in a first mode in which an external electronic device can be controlled in a virtual environment of the electronic device by an electronic device, an external Displaying a virtual external electronic device corresponding to the external electronic device based on appearance information and function information of the electronic device, and when an input event is detected in the virtual external electronic device by the electronic device, the external electronic device Transmitting input information related to the input event to a device, performing a function corresponding to the input information when the input information is received from the electronic device by the external electronic device, and a function corresponding to the input information Transmitting output information related to the state of the external electronic device changed according to the execution of the external electronic device to the electronic device, and when output information is received from the external electronic device by the electronic device, the virtual external device is based on the output information. An operation of changing the state of the electronic device to correspond to the changed state of the external electronic device.

A method for controlling an external electronic device in the real world in a virtual environment in an electronic device according to various embodiments includes appearance information and function information of the external electronic device in a first mode capable of controlling the external electronic device in the virtual environment. displaying a virtual external electronic device based on the virtual external electronic device, and when the virtual external electronic device detects the occurrence of an input event, transmits input information related to the input event to the external electronic device, and transmits input information related to the input event from the external electronic device to the external electronic device. displaying the state of the virtual external electronic device in correspondence with the changed state of the external electronic device based on the output information when output information related to the state of the external electronic device changed according to the execution of the corresponding function is received; and when output information related to an event generated in the external electronic device is received from the external electronic device, changing a state of the virtual external electronic device to correspond to a state of the external electronic device changed based on the output information. can include

According to various embodiments, a user using a virtual environment can have an uninterrupted and continuous experience by performing all forms, functions, and methods of use of an electronic device used in the real world in the same or corresponding manner as in the real world. . In addition, the user experience that integrates the virtual environment and the real world can provide a realistic virtual environment to users using the virtual environment.

1 is a block diagram of an electronic device in a network environment according to various embodiments.
2A to 2B are diagrams illustrating a system capable of controlling an external electronic device in the real world in a virtual environment according to various embodiments.
3 is a block diagram of an electronic device according to various embodiments.
4 is a block diagram of a virtual external electronic device control module of an electronic device according to various embodiments of the present disclosure;
5 is a block diagram of an external electronic device according to various embodiments.
FIG. 6 is a diagram for explaining an operation of generating outer shape information of an external electronic device in an electronic device according to various embodiments of the present disclosure.
7 is a diagram for explaining an operation of providing a virtual external electronic device to a virtual environment in an electronic device according to various embodiments of the present disclosure.
8 is a diagram illustrating a message format transmitted from an electronic device to an external electronic device according to various embodiments.
9 is a diagram for explaining an operation of generating function information of an external electronic device in an electronic device according to various embodiments of the present disclosure.
10 is a flowchart illustrating an operation of generating information for displaying a virtual external electronic device in a virtual environment in an electronic device according to various embodiments.
11 is a flowchart illustrating an operation for controlling an external electronic device in the real world in a virtual environment in an electronic device according to various embodiments of the present disclosure.
12 is a flowchart illustrating an operation for controlling an external electronic device in the real world in a virtual environment in a system according to various embodiments of the present disclosure.
13A and 13B are flowcharts for explaining an operation for controlling an external electronic device in the real world in a virtual environment in a system according to various embodiments of the present disclosure.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 180 or communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor set to detect a touch or a pressure sensor set to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, a legacy communication module). It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 is a peak data rate for eMBB realization (eg, 20 Gbps or more), a loss coverage for mMTC realization (eg, 164 dB or less), or a U-plane latency for URLLC realization (eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2A to 2B are diagrams 200a to 200b illustrating a system capable of controlling an external electronic device in the real world in a virtual environment according to various embodiments.

Referring to FIG. 2A , a system capable of controlling an external electronic device in the real world in a virtual environment according to various embodiments may include an electronic device 210 and an external electronic device 310 .

According to various embodiments, the electronic device 210 provides a virtual external electronic device corresponding to the external electronic device 310 in the real world in a first mode capable of controlling the external electronic device in a virtual environment, and Based on the communication connection with the device 310, the external electronic device 310 and information (eg, input information and output information) can be reflected to the external electronic device 310 correspondingly to operation information of the virtual external electronic device. can transmit and receive

According to an embodiment, the electronic device 210 includes a wearable electronic device, and may include, for example, a Head Mounted Display (HMD) and/or wearable glasses.

According to an embodiment, the electronic device 210 may be connected to the external electronic device 310 through short-range communication.

According to an embodiment, the electronic device 210, in the first mode, selects an external electronic device 310 from a list including at least one external electronic device stored in the memory of the electronic device 210. In order to display as a virtual external electronic device, external information of the external electronic device and function information of the external electronic device matched with identification information of the external electronic device and stored in the memory of the electronic device 210 may be detected.

According to an embodiment, the electronic device 210, in the first mode, the external electronic device 310 selected by the user from a list including at least one external electronic device stored in the memory of the electronic device 210. ) as a virtual external electronic device, a request may be made to the external electronic device 310 through which communication is connected to receive form information and/or function information of the external electronic device.

According to an embodiment, the electronic device 210, in the first mode, uses a rendering technology (eg, graphic processing (Graphic Processing) may render and display a virtual external electronic device similar to the external electronic device 310 in the real world.

According to an embodiment, when the electronic device 210 detects input information based on a user's input (eg, a user's gesture) to the virtual external electronic device in the first mode, the electronic device 210 transmits the input information through a communication connection. It can be transmitted to the external electronic device 310. The electronic device 210 outputs output information related to a state (eg, a display state and/or a sound state) of the external electronic device 310 that is changed according to the execution of a function corresponding to the input information from the external electronic device 310. Upon reception, based on the output information, the state of the virtual external electronic device may be changed to correspond to the changed state of the external electronic device and displayed.

According to an embodiment, when the electronic device 210 receives output information related to an event generated in the external electronic device from the external electronic device 310 through which communication is connected in the first mode, the output information Based on this, the state of the virtual external electronic device may be changed and displayed to correspond to the state of the external electronic device changed due to the occurrence of the event.

According to an embodiment, in the first mode, the electronic device 210 may transmit/receive information with the external electronic device 310 using a predefined message format.

According to various embodiments of the present disclosure, the external electronic device 310 may reflect manipulation information of a virtual external electronic device displayed in the first mode of the electronic device 210 to the external electronic device 310 to correspond to the electronic device 310. Information (eg, input information and output information) can be transmitted and received with the electronic device 210 based on the communication connection with the 210 .

According to an embodiment, the electronic device 210 may include a portable communication device (eg, a smart phone), a computer device, a portable medical device, or a home appliance.

According to an embodiment, the external electronic device 310, according to a request of the electronic device 210 with which communication is connected, external information of the external electronic device stored in the memory of the external electronic device 310 and/or an external electronic device Device function information may be transmitted to the electronic device 210 .

According to an embodiment, the external electronic device 310 performs a function corresponding to the input information based on input information received from the electronic device 210, and the external electronic device 310 is changed according to the execution of the function. Output information related to the state of the electronic device may be generated and transmitted to the electronic device 210 .

According to an embodiment, when an event occurs in the external electronic device 310, the external electronic device 310 generates output information related to the state of the external electronic device changed by the occurrence of the event, and Output information may be transmitted to the electronic device 210 .

According to an embodiment, the external electronic device 310 may transmit/receive information with the electronic device 210 using a predefined message format.

Referring to FIG. 2B , a system capable of controlling an external electronic device in the real world in a virtual environment according to various embodiments may include an electronic device 210, an external electronic device 310, and a server 410. there is.

According to various embodiments, the electronic device 210 provides a virtual external electronic device corresponding to the external electronic device 310 in the real world in a first mode capable of controlling the external electronic device in a virtual environment, and Information (eg, input information and output information) may be transmitted and received with the external electronic device 310 through the server 410 so that operation information of the electronic device may be reflected to the external electronic device 310 correspondingly.

According to an embodiment, the electronic device 210 includes a wearable electronic device, and may include, for example, a Head Mounted Display (HMD) and/or wearable glasses.

According to an embodiment, the electronic device 210 may be connected to the server 410 through an authentication process.

According to an embodiment, the electronic device 210, in the first mode, the external electronic device 310 selected by the user from a list including at least one external electronic device stored in the memory of the electronic device 210. ) as a virtual external electronic device, the external electronic device's external appearance information and function information of the external electronic device that are matched with the identification information of the external electronic device and stored in the memory of the electronic device 210 may be detected. .

According to an embodiment, the electronic device 210, in the first mode, includes an external electronic device 310 selected by a user from a list included in at least one external electronic device stored in the memory of the electronic device 210. ) as a virtual external electronic device, a request may be made to the server 410 to receive form information and/or function information of the external electronic device from the server 410 .

According to an embodiment, the electronic device 210, in the first mode, uses a rendering technology (eg, graphic processing (Graphic Processing) may render and display a virtual external electronic device similar to the external electronic device 310 in the real world.

According to an embodiment, when the electronic device 210 detects input information based on a user's input to the virtual external electronic device 310 in the first mode, the electronic device 210 transmits the input information through the server 410. It can be transmitted to the external electronic device 310. When the electronic device 210 receives output information related to the state of the external electronic device changed according to the execution of the function corresponding to the input information from the external electronic device 310 through the server 410, the Based on the output information, the state of the virtual external electronic device may be changed to correspond to the changed state of the external electronic device and displayed.

According to an embodiment, the electronic device 210 outputs information related to an event generated in the external electronic device 310 from the external electronic device 310 through the server 410 in the first mode. is received, the state of the virtual external electronic device may be changed and displayed to correspond to the state of the external electronic device 310 changed by the occurrence of the event based on the output information.

According to an embodiment, in the first mode, the electronic device 210 may transmit/receive information with the external electronic device 310 through the server 410 using a predefined message format.

According to various embodiments of the present disclosure, the external electronic device 310 may operate on a server so that operation information of a virtual external electronic device displayed in the first mode of the electronic device 210 may be reflected to the external electronic device 310 correspondingly. Information (eg, input information and output information) can be transmitted and received with the electronic device 210 through 410).

According to an embodiment, the external electronic device 310 may be connected to the server 410 through an authentication process.

According to an embodiment, the external electronic device 310 performs a function corresponding to the input information based on input information received from the electronic device 210 through the server 410, and the function Output information related to the state of the external electronic device changed according to the execution of is transmitted to the electronic device 210 through the server 410 .

According to an embodiment, when an event occurs in the external electronic device 310, the external electronic device 310 changes a state (eg, a display state and/or a sound state) of the external electronic device when the event occurs. Output information related to may be generated, and the generated output information may be transmitted to the electronic device 210 through the server 410 .

According to an embodiment, the external electronic device 210 may transmit and receive information with the electronic device 210 through the server 410 using a predefined message format.

According to various embodiments of the present disclosure, the server 410 controls the electronic device 210 to reflect operation information of the virtual external electronic device displayed in the first mode of the electronic device 210 to the external electronic device 310 correspondingly. ) may transmit information (eg, input information) received from the external electronic device 310, and information (eg, output information) received from the external electronic device 310 may be transmitted to the electronic device 210.

According to an embodiment, the server 410, according to the request of the electronic device 210, external information and / or function information of the external electronic device stored in the DB of the server 410 to the It can be transmitted to the electronic device 210.

According to an embodiment, the DB of the server 410 may store external electronic device appearance information and function information of the external electronic device that match identification information of at least one external electronic device for each user account.

According to an embodiment, the server 410 transmits information (eg, input information) received from the electronic device 210 and the external electronic device 310 to the outside in the form of a predefined message. It is transmitted to the electronic device 310, and information (eg, output information) received from the external electronic device 310 may be transmitted to the electronic device 210, and the exchange of the message is performed using a predefined Application Layer protocol ) can be performed by

According to an embodiment, the protocol used when transferring information in the form of a message between the electronic device 210 and the external electronic device 310 in the server 410 is HTTP (Port number 80), information security, and forgery and tampering. To prevent this, either HTTPS (Port number 443) protocol or protocols such as QUIC or UDP for congestion control during streaming can be used.

3 is a block diagram 300 of an electronic device according to various embodiments.

Referring to FIG. 3 , according to various embodiments, an electronic device 301 (eg, the electronic device 101 of FIG. 1 and/or the electronic device 210 of FIGS. 2A and 2B ) includes a processor 320 , a memory 330, a display 360, a sensor module 376, a camera module 380, and a communication module 390.

According to various embodiments of the present disclosure, the processor 320 may, in a mode capable of registering an external electronic device in a virtual environment (eg, a second mode), generate a virtual external electronic device corresponding to at least one external electronic device existing in the real world. Appearance information and function information of an external electronic device for generating a device may be created and stored.

According to one embodiment, the processor 320, in the mode (eg, the second mode), uses a 3D object recognition and / or scanning (3D Recognition / Scanning) technology to the camera module 380 and / or Image data and/or sensor data for an external electronic device detected by the sensor module 376 (eg, the external electronic device 310 of FIGS. 2A and 2B ) is analyzed, and a 3D image is created based on the analyzed data. Coordinate information, texture, and shader data may be generated, and the generated 3D coordinate information, texture, and shader data may be generated as exterior information of the external electronic device.

According to an embodiment, the processor 320 may, in the mode (eg, the second mode), send a graphic processing unit (eg, a graphic processing unit (GPU)) included in the electronic device 301 to the external electronic device. While a virtual external electronic device is rendered and displayed by applying external appearance information of the device, a function setting menu (eg, a GUI menu) for setting functions of the virtual external electronic device may be displayed. The processor 320 checks the type of at least one function selected by the user through the function setting menu, and the area where a component for performing at least one function in the displayed virtual external electronic device is to be located. If this is selected by the user, the area selected by the user can be identified as area information of the component for performing the function. The processor 320 may generate information about the type of the function and the region of the component for performing the function as function information of the external electronic device.

According to an embodiment, the processor 320, in the mode (eg, the second mode), in a state in which short-range communication is connected with the external electronic device, identifies the external electronic device from the external electronic device (eg, the second mode). ID information of the external electronic device) may be received, and appearance information and function information of the external electronic device may be stored in the memory 330 by matching with the identification information of the external electronic device.

According to an embodiment, the processor 320, in the mode (eg, the second mode), when identification information (eg, id information of the electronic device) of the external electronic device input by the user is input, the external electronic device Appearance information and function information of the external electronic device may be stored in the memory 330 by matching with the identification information of the electronic device.

According to an embodiment, the processor 320, in the mode (eg, the second mode), in a state in which short-range communication is connected with the external electronic device, identifies the external electronic device from the external electronic device (eg, the second mode). ID information of the electronic device), appearance information of the external electronic device, and function information of the external electronic device are received, and the external electronic device's appearance information and function information are stored in the memory 330 by matching with the identification information of the external electronic device. can be saved

According to an embodiment, the processor 320, in the mode (eg, the second mode), uses a function setting menu (eg, GUI menu) including functions previously stored in the electronic device 301 to configure the Functional information of an external electronic device may be generated.

According to an embodiment, the processor 320 generates function information of the external electronic device using a function setting menu (eg, a GUI menu) including functions received from the external electronic device to which short-range communication is connected. can

According to an embodiment, the processor 320 uses a function setting menu (eg, a GUI menu) including functions received from a server (eg, the server 410 of FIG. 2B) to perform functions of the external electronic device. information can be generated.

According to an embodiment, the processor 320 transmits the identification information of the external electronic device, the appearance information of the external electronic device, and the function information of the external electronic device to the server and stores them in a DB of the server. .

According to various embodiments, the processor 320 renders and displays a virtual external electronic device corresponding to an external electronic device in the real world in a mode (eg, a first mode) capable of controlling an external electronic device in a virtual environment. can do.

According to an embodiment, the processor 320 may provide a virtual environment by executing an application for executing virtual reality (VR) or an application for executing virtual reality (VR) (e.g. : first mode), and in the mode (eg, first mode), appearance information and function information of the external electronic device selected by the user may be detected.

According to an embodiment, the processor 320, in the mode (eg, the first mode), determines the appearance of an external electronic device selected by the user from a list including at least one external electronic device stored in the memory 330. Information and function information can be detected from the memory 330 .

According to an embodiment, the processor 320, in the mode (eg, the first mode), determines the appearance of an external electronic device selected by the user from a list including at least one external electronic device stored in the memory 330. Information and function information may be requested from the server and received from the server.

For example, the processor 320, in order to request appearance information and function information of the external electronic device selected by the user, to the server, as in Example 1 below, the user's account information (UserID), external A message including identification information (Device ID) of the electronic device and a security key (MagicKey) may be transmitted to the server.

<Example 1>

For example, the processor 320, in response to a request for external appearance information and function information of the virtual external electronic device, obtains external external electronic device external information (objectDataVertex) and external electronic device from the server as shown in <Example 2> below. A message in the form of a packet including feature information (objectFeature) of can be received.

<Example 2>

According to an embodiment, the processor 320, in the mode (eg, the first mode), external information (eg, 3D coordinate information, texture and shader data) and functions of the external electronic device A virtual external electronic device similar to the external electronic device 310 in the real world may be rendered and displayed by reflecting the information to a rendering technology (eg, graphic processing) constituting the virtual environment.

According to an embodiment, the processor 320, in the mode (eg, the first mode), calculates a light source in addition to external shape information (eg, 3D coordinate information, texture and shader data) of the external electronic device. and a 3D graphics API (e.g. DirectX, Opengl, Vulkan and/or Metal) to which the external information and additional information (e.g. light source, etc.) are applied to create a 3D virtual external electronic device through the GPU pipeline. It can be visualized graphically. In the process of shaping the virtual external electronic device into the 3-resource graphics, the following <Example 3> of graphics processing unit (GPU) rendering using vertex and shader is the same.

<Example 3>

According to an embodiment, the processor 320 may, in the mode (eg, the first mode), based on function information of the external electronic device, an area where a component for performing a function in a virtual external electronic device is to be located. can be specified.

For example, in the mode (eg, the first mode), the processor 320 checks a hinge function for folding the external electronic device, a screen output function, and a touch function with function information of the external electronic device. Hinge configuration for performing a hinge function capable of folding the external electronic device based on function information of the external electronic device after rendering the external appearance of the virtual external electronic device based on the external electronic device external information. A hinge area where an element is located, a display area where a display component for performing a screen output function is located, and an input area where a display component or button component for performing a touch function are located may be identified.

In the case of a hinge function capable of folding and unfolding the virtual external electronic device around the hinge in the mode (eg, the first mode), the processor 320 sets the center point of the virtual external electronic device as a hinge A vertex shader can be added to specify a coordinate (e.g., hinge area) and perform a rotational geometric transformation based on the coordinate. When the processor 320 calculates assuming that the axis of the hinge displayed in the hinge area of the virtual external electronic device is the Z axis, the processor 320 uses a graphic application programming interface (API) such as Rotate and/or Transform to generate the virtual external electronic device. Folding and unfolding of a virtual external electronic device may be implemented and displayed in a virtual environment by reflecting Equation 1 below for coordinate information according to folding and unfolding of .

<Formula 1>

In the mode (eg, the first mode), in the case of a screen output function, the processor 320 may designate a display area of the virtual external electronic device as an area to be texture mapped. Thereafter, the processor 320 controls the virtual external electronic device and the real world external electronic device (eg, the external electronic device 310 of FIGS. 2A and 2B ) to correspond to each other, from the real world external electronic device. Based on the received output information, the screen state (eg, screen image) corresponding to (or identical to) the screen state (eg, screen image) of the display of the external electronic device is mapped to the display area of the virtual external electronic device. ) can be done. The processor 320 stores a corresponding texture (eg, a screen image) in a memory shared with a GPU (Graphic Processing Unit) to increase calculation processing speed, and when texture mapping is performed, the screen image is copied to the memory. You can bring the effect of reflecting the texture (e.g. screen image) without any calculation. The screen image is stored in a memory shared with a GPU (Graphic Processing Unit) and texture-mapped as shown in <Example 4> below.

<Example 4>

In the mode (eg, the first mode), in the case of a button function, the processor 320 may designate a display area or a button area as a touch event processing area and recognize a touch interaction by a user's gesture. .

According to various embodiments of the present disclosure, in a mode capable of controlling an external electronic device in a virtual environment (eg, a first mode), the processor 320 converts operation information of a displayed virtual external electronic device to the external electronic device (eg, a first mode). Information (eg, input information and output information) may be transmitted and received with the external electronic device so as to correspond to the external electronic device 310 of FIGS. 2A and 2B .

According to an embodiment, when the processor 320 detects the occurrence of an input event in the virtual external electronic device while displaying the virtual external electronic device in the mode (eg, the first mode), the input event It is possible to generate input information related to and transmit the generated input information to the external electronic device.

For example, in the mode (eg, the first mode), the processor 320 may generate an input event (or touch event) in a display area or a button area of the virtual external electronic device based on a user's gesture. If detected, input information including the type of input, location of input, number of times and duration of input may be generated as a message and transmitted to the external electronic device as shown in Example 5 below.

<Example 5>

According to an embodiment, the processor 320 is configured to perform a function corresponding to the input information from the external electronic device according to transmission of a message including the input information in the mode (eg, the first mode). When receiving output information related to the state of the external electronic device changed according to the output information, the state of the virtual external electronic device (eg, display area state and/or sound state) is compared with the changed state of the external electronic device based on the output information. can be changed accordingly.

According to an embodiment, when the processor 320 receives a message including output information related to an event generated in the external electronic device from the external electronic device in the mode (eg, the first mode), the Based on the output information, a state (eg, a display area state and/or a sound state) of the virtual external electronic device may be changed to correspond to the state of the external electronic device changed by the occurrence of the event.

According to an embodiment, the processor 320, in the mode (eg, the first mode), when the electronic device 301 is connected to the external electronic self-government through the short-range communication module of the communication module 390 , A message including information (eg, input information and/or output information) may be transmitted and received with the external electronic device.

According to an embodiment, the processor 320, in the mode (eg, the first mode), when the electronic device 301 is not connected to the external electronic self-government through the short-range communication module of the communication module, Alternatively, according to a user's selection, a message including information (eg, input information and/or output information) may be transmitted and received with the external electronic device through a server (eg, the server 410 of FIG. 2B ).

According to one embodiment, the processor 320, in the mode (eg, the first mode), requests access to the server based on the user's account information (eg, id and password), and If authentication is successful through the authentication process, information can be transmitted and received with the external electronic device through the server.

According to an embodiment, a message type including information (eg, input information and/or output information) transmitted to and received from the external electronic device directly or through the server is the external electronic device or the external electronic device. It can include the message format of the server and the predefined method.

For example, when a call is received from an external electronic device in the real world (eg, the external electronic device 310 of FIG. 2B ), the external electronic device provides a frame buffer for a call reception screen and a sound stream for a ringtone. When capturing and generating output information and transmitting a message including the output information in a stream form to a server (eg, the server 410 of FIG. 2B ), the server transmits a message including the output information to the electronic device ( 301). In the mode (eg, the first mode), the processor 320 of the electronic device 301 performs texture mapping on the display area of the virtual external electronic device based on the received output information to perform texture mapping on the virtual external electronic device. As a call reception screen notifying the reception of a call is displayed on the display of the electronic device and a bell sound announcing the reception of a call is output through the speaker of the electronic device, the user receives a call from an external electronic device in the real world in a virtual environment It can be seen that there is a state

The processor 320, in the mode (eg, the first mode), when detecting an input event in which a call button is input (or touched) by a user's gesture, generates input information related to the input of the call button, , A message including the generated input information may be transmitted to the external electronic device through the server.

The external electronic device, based on the received input information, converts to a call state in which a call button of the external electronic device is input, captures a frame buffer for the call connection state screen, and generates output information; When a message including the output information is transmitted to the server in the form of a stream, the server may transmit the message including the output information to the electronic device 301 . In the mode (eg, the first mode), the processor 320 of the electronic device 301 performs texture mapping on the display area of the virtual external electronic device based on the received output information to perform texture mapping on the virtual external electronic device. As a call connection screen indicating a call connection state is displayed on the display screen of the electronic device, the user can know that a call is connected to an external electronic device in the real world in the virtual environment. Thereafter, the external electronic device transmits output information obtained by capturing a sound stream of a caller's voice output through a speaker of the external electronic device to the electronic device 301 through the server, and the electronic device 301 transmits the output information to the electronic device 301 through the server. As the output information obtained by capturing the sound stream of the user's voice input through the microphone of the electronic device is transmitted to the external electronic device through the server, the user can make a call using the virtual external electronic device in a virtual environment. can

According to various embodiments, the processor 320 may register a virtual external electronic device corresponding to at least one external electronic device existing in the real world in a mode (eg, a second mode) capable of registering an external electronic device in a virtual environment. Manipulation of the virtual external electronic device displayed in a mode (eg, the first mode) capable of generating information (eg, external electronic device appearance information and function information) for generation and controlling the external electronic device in a virtual environment. A virtual external electronic device control module capable of reflecting information correspondingly to the external electronic device 301 is included, or the virtual external electronic device control module is stored as a software module in the memory 330 to control the processor 320 It may operate under, or may be separately configured in the electronic device 301. The configuration of the control module of the virtual external electronic device can be described in detail with reference to FIG. 4 below.

According to various embodiments, the memory 330 may be implemented substantially the same as or similar to the memory 130 of FIG. 1 .

According to an embodiment, identification information of at least one external electronic device and external information and function information of the external electronic device corresponding to the identification information of the at least one external electronic device may be stored in the memory 330 .

According to various embodiments, the memory 360 may be implemented substantially the same as or similar to the display 360 of FIG. 1 .

According to an embodiment, in a first mode capable of controlling an external electronic device in a virtual environment through the display 360, at least one virtual external electronic device may be displayed.

According to various embodiments, the sensor module 376 may be implemented substantially the same as or similar to the sensor module 376 of FIG. 1 .

According to an embodiment, the sensor module 376 may include an angle sensor, a tilt sensor, an acceleration sensor, a geomagnetic sensor, a gyro sensor, a motion sensor, a gravity sensor, and a proximity sensor.

According to an embodiment, the sensor module 376 performs 3D object recognition and/or scanning (3D Recognition/Scanning) technology in a mode (eg, second mode) capable of registering an external electronic device in a virtual environment. An external electronic device in the real world may be detected in order to generate appearance information of a virtual external electronic device using .

According to various embodiments, the camera module 380 may be implemented substantially the same as or similar to the display module 180 of FIG. 1 .

According to an embodiment, the camera module 380 performs 3D object recognition and/or scanning (3D Recognition/Scanning) technology in a mode (eg, second mode) capable of registering an external electronic device in a virtual environment. An external electronic device in the real world may be detected in order to generate appearance information of a virtual external electronic device using .

According to various embodiments, the communication module 390 may be implemented substantially the same as or similar to the communication module 190 of FIG. 1 and may include a plurality of communication circuits using different communication technologies.

According to an embodiment, the communication module 390 may include a plurality of communication circuits, and may include at least one of a wireless LAN module (not shown) and a short-range communication module (not shown), and the short-range The communication module (not shown) may include a UWB communication module, a Wi-Fi communication module, a near field communication (NFC) communication module, a Bluetooth legacy communication module, and/or a bluetooth low energy (BLE) communication module.

4 is a block diagram 400 of a virtual external electronic device control module of an electronic device according to various embodiments.

Referring to FIG. 4 , the virtual external electronic device control module 401 controls at least one external electronic device existing in the real world in a mode (eg, second mode) capable of registering an external electronic device in a virtual environment. Information for generating a virtual external electronic device corresponding to (eg, appearance information and function information of the external electronic device) is generated and displayed in a mode (eg, first mode) capable of controlling the external electronic device in a virtual environment. Operation information of a virtual external electronic device to be used may be reflected correspondingly to the external electronic device 301 . The virtual external electronic device control module 401 is included in a processor (eg, the processor 320 of FIG. 3 ), stored as software in a memory (eg, the memory 330 of FIG. 3 ), or an electronic device ( Example: 301) of FIG. 3 may be configured separately. The virtual external electronic device control module 401 includes an object detection unit 411, a gesture detection unit 413, a physical engine unit 415, a function management unit 417, a rendering processing unit 419, and an information receiving unit 421. can include

According to various embodiments, the shape detector 411 may register a virtual external electronic device corresponding to an external electronic device in the real world in a mode (eg, a second mode) capable of registering an external electronic device in a virtual environment. Appearance information of an external electronic device for providing a device in a virtual environment may be generated and stored in a memory (eg, the memory 330 of FIG. 3 ) or transmitted to a server (eg, the server 410 of FIG. 2B ).

According to one embodiment, the shape detector 411, in the mode (eg, the second mode), uses a 3D object recognition and / or scanning (3D Recognition / Scanning) technology to the camera module ( To an external electronic device (eg, the external electronic device 310 of FIGS. 2A and 2B ) detected through the camera module 380 of FIG. 3 ) and/or a sensor module (eg, the sensor module 376 of FIG. 3 ) Image data and/or sensor data may be analyzed. The appearance detection unit 411 generates 3D coordinate information, texture and shader data based on the analyzed data, and generates the generated 3D coordinate information, texture and shader data as appearance information of an external electronic device. can

According to various embodiments, the gesture detection unit 413 may control a camera module (the camera module 380 of FIG. 3 ) and/or a sensor in a mode capable of controlling an external electronic device in a virtual environment (eg, the first mode). A function of recognizing a user's gesture may be performed through a module (eg, the sensor module 376 of FIG. 3 ).

According to one embodiment, the gesture detection unit 413, in the mode (eg, the first mode), a camera module (eg, the camera module 380 of FIG. 3) and/or a sensor module (eg, the sensor module of FIG. 3) (376)) to determine the location information of the user's body part (e.g., hand) where the gesture is performed, convert the checked location information into location information in the virtual environment, and convert the user's gesture in the real world to the virtual environment. You can switch to gestures in .

According to various embodiments, the physics engine unit 415 may perform a function of reflecting a user gesture in the real world to the virtual environment in a mode capable of controlling an external electronic device in a virtual environment (eg, a first mode). can

According to an embodiment, the physical engine unit 415 uses the user's gesture in the virtual environment detected by the gesture detection unit 143 to perform a physical operation (eg, button input) on a virtual external electronic device displayed in the virtual environment. ) can perform the reflecting function.

According to various embodiments of the present disclosure, the function manager 417, in a mode capable of registering an external electronic device in a virtual environment (eg, the second mode), stores a virtual external electronic device corresponding to an external electronic device in the real world in a virtual environment. Functional information of an external electronic device to be provided in may be generated and stored in a memory (eg, the memory 330 of FIG. 3 ) or transmitted to a server (eg, the server 410 of FIG. 2B ).

According to an embodiment, the function management unit 417 generates input information when an input event is detected in a virtual external electronic device displayed in a mode capable of controlling an external electronic device in a virtual environment (eg, a first mode). And, a message including the generated input information may be generated and transmitted to the external electronic device.

According to various embodiments, the rendering processing unit 419 may, in a mode capable of controlling the external electronic device in a virtual environment (eg, the first mode), generate external electronic device information and A function of the external electronic device created by the function management unit 415 may be rendered and displayed as a virtual external electronic device based on information.

According to an embodiment, the rendering processing unit 419, in the first mode, uses a rendering technology (eg, graphic processing )), a virtual external electronic device similar to the external electronic device 310 in the real world may be rendered and displayed.

According to an embodiment, the rendering processing unit 419 uses a 3D API (DirectX, OpenGL, Vulkan, Metal) in the first mode, and based on the shape information generated by the shape detector 411 This can be done through rendering processes such as mesh construction and texture mapping. The rendered virtual external electronic device is output through an electronic device (eg, a head mount device (HMD) worn by the user) so that the user can visually see it.

According to an embodiment, the rendering processing unit 419, in addition to visually outputting the virtual external electronic device, outputs "sound" through the speaker of the electronic device, and creates a "glove-type" and/or "cloth-type" tactile device that interlocks with the electronic device. Through this, 　 can be conveyed to the user tactilely.

According to an embodiment, the rendering processor 419, in the first mode, based on the function information of the external electronic device generated by the function management unit, according to the interaction in the virtual environment object (eg, virtual external electronic device) A coordinate value, color, and/or light of an area where an input event is detected in a device) may be calculated and reflected in rendering of a virtual external electronic device as a parameter of 3D operation.

According to various embodiments, the information receiving unit 421 may receive output information (eg, screen data) received from an external electronic device in the real world in a mode (eg, a first mode) capable of controlling an external electronic device in a virtual environment. and/or sound data) may perform a function of receiving.

According to an embodiment, the information receiving unit 421 displays the external electronic device in a display area of a virtual external electronic device based on the output information received from the external electronic device in the mode (eg, the first mode). It can be controlled to output a screen corresponding to (or the same as) the screen of the display of and to output the same (or similar) sound output from the external electronic device.

5 is a block diagram 500 of an external electronic device according to various embodiments.

Referring to FIG. 5, according to various embodiments, an external electronic device 501 (eg, the electronic device 101 of FIG. 1 and/or the external electronic device 310 of FIGS. 2A to 2B) may include a processor ( 520), a memory 530, a display 560, and a communication module 590.

According to various embodiments, the processor 520 may control an external electronic device in a virtual environment of an electronic device (eg, the electronic device 210 of FIGS. 2A to 2B or the electronic device 301 of FIG. 3 ). The electronic device (eg, the electronic device 210 of FIGS. 2A to 2B ) and/or the electronic device (eg, the electronic device 210 of FIGS. Based on the request of the electronic device 301 in Step 3, identification information of the external electronic device or identification information, appearance information, and function information of the external electronic device may be transmitted to the electronic device.

According to an embodiment, the processor 520, in the mode (eg, the first mode or the second mode), in a state in which communication is connected with the electronic device based on short-range communication of the communication module 590, Information (eg, identification information of the external electronic device or identification information of the external electronic device, appearance information, and function information) may be transmitted to the electronic device.

According to an embodiment, the processor 520, in the mode (eg, the first mode or the second mode), based on the short-range communication of the communication module 290 in a state in which communication with the electronic device is not connected , Information (eg, identification information of an external electronic device or identification information of the external electronic device, appearance information, and function information) may be transmitted to the electronic device through a server (eg, the server 410 of FIG. 2B ).

According to various embodiments, the processor 520 may control an external electronic device in a virtual environment of an electronic device (eg, the electronic device 210 of FIGS. 2A to 2B or the electronic device 301 of FIG. 3 ). Transmits and receives information (eg, input information and/or output information) with the electronic device so that operation information of the virtual external electronic device displayed in the mode (eg, first mode) can be reflected correspondingly to the external electronic device 501 can do.

According to an embodiment, when an event occurs in the external electronic device 501 and the state of the external electronic device is changed, the processor 520 generates the output information related to the changed state of the external electronic device, It can be transmitted to the electronic device.

For example, the processor 520 detects the occurrence of a call reception event in the external electronic device 501, and displays a call reception screen displayed on the display 560 of the external electronic device according to the call reception event. A frame buffer for a frame buffer and a sound stream for a ring tone indicating conversion reception may be captured and generated as output information, and a message including the generated output information may be transmitted to the electronic device.

According to an embodiment, when input information is received from the electronic device, the processor 520 performs a function corresponding to the input information, and the external electronic device changed according to the execution of the function corresponding to the input information. State-related output information may be transmitted to the electronic device.

For example, when the processor 520 receives input information related to inputting a call button from the electronic device, the processor 520 switches to a call state in which the call button of the external electronic device has been input, and displays information on the call connection state screen. A frame buffer may be captured, generated as output information, and a message including the output information may be transmitted to the electronic device 301 in the form of a stream.

According to an embodiment, the processor 520, in a state in which communication with the electronic device is connected based on short-range communication of the communication module 590, information (eg, input information and/or output information) with the electronic device It is possible to transmit and receive messages including.

According to an embodiment, the processor 520, in a state in which communication with the electronic device is not connected based on short-range communication of the communication module 590, through a server (eg, the server 410 of FIG. 2B) A message including information (eg, input information and/or output information) may be transmitted and received with the electronic device.

According to various embodiments, the memory 530 may be implemented substantially the same as or similar to the memory 130 of FIG. 1 .

According to an embodiment, the memory 530 may store identification information of an external electronic device.

According to an embodiment, the memory 530 may store identification information of the external electronic device, external information of the external electronic device, and function information of the external electronic device.

According to various embodiments, the display 560 may be implemented substantially the same as or similar to the display 360 of FIG. 1 .

According to an embodiment, a screen corresponding to an event occurring in the external electronic device 501 may be displayed through the display 560 .

According to various embodiments, the communication module 590 may be implemented substantially the same as or similar to the communication module 190 of FIG. 1 and may include a plurality of communication circuits using different communication technologies.

According to an embodiment, the communication module 590 may include a plurality of communication circuits, and may include at least one of a wireless LAN module (not shown) and a short-distance communication module (not shown). The communication module (not shown) may include a UWB communication module, a Wi-Fi communication module, a near field communication (NFC) communication module, a Bluetooth legacy communication module, and/or a bluetooth low energy (BLE) communication module.

FIG. 6 is a diagram 600 for explaining an operation of generating outer shape information of an external electronic device in an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 6 , the electronic device 210 (eg, the electronic device 301 of FIG. 3 ), in a mode (eg, the second mode) capable of registering an external electronic device in a virtual environment, the electronic device When the external electronic device 310 in the real world is detected through the camera module (eg, the camera module 380 of FIG. 3) of 210, 3D object recognition and/or scanning (3D Recognition/Scanning) technology is used. Thus, image data and/or sensor data of the detected external electronic device 310 may be analyzed. The electronic device 210 generates 3D coordinate information, texture and shader data based on the analyzed data, and generates the 3D coordinate information, texture and shader data as external information of the external electronic device, It can be stored in the memory (eg, the memory 230 of FIG. 3) of the electronic device.

7 is a diagram 700 for explaining an operation of providing a virtual external electronic device to a virtual environment in an electronic device according to various embodiments.

Referring to FIG. 7 , the electronic device 210 is selected by a user from a list including at least one external electronic device in a mode capable of controlling an external electronic device in a virtual environment (eg, a first mode). A virtual external electronic device 711 may be rendered based on the appearance information and function information of the external electronic device and displayed on the display 760 of the electronic device 210 providing the virtual environment.

The electronic device 201 checks the positional information of the user's body part (eg, hand) where the gesture is performed through the camera module (the camera module 380 of FIG. 3) of the electronic device 210, and checks the location information. The obtained location information may be converted into location information in the virtual environment, so that the user gesture 713 in the real world may be converted into a user gesture 713a in the virtual environment. The electronic device 210 calculates a physical interaction between the user gesture 713a in the virtual environment and the virtual external electronic device 711, reflects the result of the calculation, and renders the virtual external electronic device 711. can be displayed.

8 is a diagram 800 illustrating a message format transmitted from an electronic device to an external electronic device according to various embodiments.

Referring to FIG. 8, an electronic device (eg, the electronic device 210 of FIGS. 2A to 2B and/or the electronic device 301 of FIG. 3) may control an external electronic device in a virtual environment ( Example: In the first mode), when an input event (or touch event) is detected in the display area or button area of the virtual external electronic device based on the user's gesture, the type of input, the location of the input, and the number of inputs It is possible to generate a message 810 including input information including a time and a duration. A plurality of header information (eg, data link header (DH), network header (NH), transport header (TH), session header (SH), and physical header (PH)) is included in the header field 811 of the message 810. can capture. The content area 813 of the message 810 may include input type information 813a, input location, input frequency, and input duration information 813b. The electronic device may transmit the message 810 to an external electronic device (eg, the external electronic device 310 of FIGS. 2A and 2B).

9 is a diagram 900 for explaining an operation of generating function information of an external electronic device in an electronic device according to various embodiments.

Referring to FIG. 9, an electronic device (eg, the electronic device 210 of FIGS. 2A to 2B and/or the electronic device 301 of FIG. 3) may register an external electronic device in a virtual environment ( Example: In the second mode), a virtual external electronic device 911 is rendered and displayed by applying appearance information of the external electronic device to a graphic processing unit (eg, GPU (Graphic Process Unit)) included in the electronic device. During this time, a function setting menu (eg, GUI menu) 930 for setting functions of the virtual external electronic device may be displayed.

After the user selects the add hinge function 931 among the functions provided by the menu 930 (eg, add hinge, add display, and add buttons), the electronic device controls the user's operation in the virtual external electronic device. If size and/or location information on the hinge area 913 set by (eg, a drag operation) is checked, information on the hinge area that can perform the hinge function and folding/unfolding of the electronic device can be generated. .

When the user selects the display addition function 933 among the functions provided by the menu 930 (eg, adding a hinge, adding a display, and adding a button), the electronic device provides a sub screen for selecting the aspect ratio of the display. A menu 933a (eg, 4:3 ratio and 16:9 ratio) may be provided. After the display addition function 933 is selected by the user, the electronic device determines the size and/or size of the display area 916 set by the user's operation (eg, a drag operation) on the virtual external electronic device 911. When the location information is confirmed, the display function and area information of the display for screen output may be generated as function information of the external electronic device.

In the electronic device, when a button addition function 935 is selected by the user among the functions provided by the menu 930 (eg, adding a hinge, adding a display, and adding a button), a submenu capable of selecting the type of button 935a (eg, sound button and power button) may be provided. After the button addition function 935 is selected by the user, the electronic device checks the size and/or location information of the button area 919 set by the user's operation in the virtual external electronic device, the button function And button region information for outputting an input event (eg, a touch event) may be generated as function information of an external electronic device.

According to various embodiments of the present disclosure, a system capable of controlling an external electronic device in the real world in a virtual environment, in a first mode capable of controlling an external electronic device in a virtual environment, based on appearance information and function information of the external electronic device. A virtual external electronic device corresponding to the external electronic device is displayed, and when the occurrence of an input event is detected in the virtual external electronic device, input information related to the input event is transmitted to the external electronic device. When output information related to the state of the external electronic device changed according to the execution of the function corresponding to the input information is received from the virtual external electronic device, the state of the virtual external electronic device corresponds to the changed state of the external electronic device based on the output information. When receiving the input information from the electronic device (e.g., the electronic device 301 of FIG. 3) and the electronic device, the function corresponding to the input information is performed, and the function corresponding to the input information is performed. An external electronic device (eg, the electronic device 501 of FIG. 5 ) that transmits output information related to the state of the external electronic device changed according to the change to the electronic device may be included.

According to various embodiments of the present disclosure, the electronic device recognizes the external electronic device in a second mode capable of registering the external electronic device in a virtual environment, and changes the appearance of the external electronic device based on the recognition of the external electronic device. information is generated, a virtual external electronic device generated based on the appearance information and a function setting menu capable of setting functions of the virtual electronic device are displayed, and the type of at least one function selected through the function setting menu and the The virtual external electronic device may be set to generate information about a region where a component for performing the at least one function is located as the function information, and to store the appearance information and the function information to match the external electronic device. .

According to various embodiments, at least one function included in the function setting menu may be pre-stored in the electronic device, received from the external electronic device, or set to be received from a server.

According to various embodiments of the present disclosure, when the electronic device receives output information related to an event generated in the external electronic device from the external electronic device in the first mode, the electronic device changed to the occurrence of the event based on the output information. It may be set to change the state of the virtual external electronic device to correspond to the state of the external electronic device.

According to various embodiments, when an event occurs in the external electronic device and the state of the external electronic device is changed, the external electronic device generates the output information related to the changed state of the external electronic device to the electronic device. can be set to transmit.

According to various embodiments, when short-range communication is connected with the external electronic device, the electronic device may be set to transmit/receive information with the external electronic device based on the short-range communication connection.

According to various embodiments, the electronic device may be set to transmit/receive information of the external electronic device through a server when short-range communication is not connected with the external electronic device.

According to various embodiments, an electronic device capable of controlling an external electronic device in the real world in a virtual environment includes a camera module (eg, the camera module 380 of FIG. 3 ) and a processor (eg, the processor 320 of FIG. 3 ). )), wherein the processor displays a virtual external electronic device based on appearance information and function information of the external electronic device in a first mode capable of controlling the external electronic device in a virtual environment, and the virtual external electronic device is displayed. When the device detects the occurrence of an input event, it transmits input information related to the input event to an external electronic device, and the state of the external electronic device changed according to the execution of a function corresponding to the input information from the external electronic device. When output information is received, the state of the virtual external electronic device is displayed to correspond to the changed state of the external electronic device based on the output information, and output information related to an event generated in the external electronic device is displayed from the external electronic device. is received, it may be set to change the state of the virtual external electronic device to correspond to the state of the external electronic device changed based on the output information.

According to various embodiments of the present disclosure, in a second mode capable of registering an external electronic device in a virtual environment, the processor recognizes the external electronic device through the camera module, and the external electronic device is recognized based on the external electronic device. External appearance information of the electronic device is generated, a virtual external electronic device created based on the external appearance information and a menu capable of setting functions of the virtual electronic device are displayed, and the type of at least one function selected through the menu and the A location of a function to be selected to perform the at least one function in the virtual external electronic device may be generated as the function information, and the appearance information and the function information may be stored to match the external electronic device.

According to various embodiments of the present disclosure, when the electronic device is in a short-range communication connection state with the external electronic device, the processor transmits and receives information with the external electronic device based on the short-range communication connection, and the electronic device communicates with the external electronic device. If the short-range communication is not connected with the device, it may be set to transmit and receive information of the external electronic device through a server.

10 is a flowchart 1000 illustrating an operation of generating information for displaying a virtual external electronic device in a virtual environment in an electronic device according to various embodiments. Operations capable of controlling an external electronic device in the real world in the virtual environment may include operations 1001 to 1007. According to an embodiment, at least one of operations 1001 to 1007 may be omitted, the order of some operations may be changed, or another operation may be added.

In operation 1001, an electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 210 of FIGS. 2A to 2B , and/or the electronic device 301 of FIG. 3 ) operates an external electronic device in a virtual environment. In a registration mode (eg, the second mode), an external electronic device (eg, the external electronic device 310 of FIGS. 2A and 2B ) in the real world may be detected.

According to an embodiment, the electronic device, in the mode (eg, the second mode), a camera module (eg, the camera module 380 of FIG. 3) or/and a sensor module (eg, the sensor module of FIG. 3 ( 376)) to detect (or obtain) the external electronic device.

In operation 1003, an electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 210 of FIGS. 2A to 2B , and/or the electronic device 301 of FIG. 3 ), an external electronic device (eg, FIG. Appearance information of the external electronic device 310 of FIGS. 2A to 2B may be generated and stored.

According to an embodiment, the electronic device, in the mode (eg, the second mode), uses a 3D object recognition or/and scanning (3D Recognition/Scanning) technology to detect the camera module or/and the sensor module. image data or/and sensor data for the external electronic device detected through the analysis are analyzed, 3D coordinate information, texture and shader data are generated based on the analyzed data, and the generated 3D coordinate information, texture and Shader data may be generated as exterior information of an external electronic device.

In operation 1005, the electronic device (eg, the electronic device 101 of FIG. 1, the electronic device 210 of FIGS. 2A to 2B, and/or the electronic device 301 of FIG. Function information of the external electronic device 310 of FIGS. 2A to 2B may be stored.

According to an embodiment, in the mode (eg, the second mode), the electronic device transmits external appearance information of the external electronic device to a graphic processing unit (eg, a graphic processing unit (GPU)) included in the electronic device. While the virtual external electronic device is rendered and displayed, a function setting menu (eg, GUI menu) for setting functions of the virtual external electronic device may be displayed. The electronic device checks the type of at least one function selected by the user through the function setting menu, and determines an area where a component for performing the at least one function is located in the virtual external electronic device by the user. If designated, the type of function and region information of a component for performing the function may be generated as function information of the external electronic device.

In operation 1007, the electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 210 of FIGS. 2A to 2B , and/or the electronic device 301 of FIG. 3 ) provides external information about the external electronic device and Function information may be stored as information for displaying a virtual external electronic device corresponding to an external electronic device in a virtual environment.

According to an embodiment, in the mode (eg, the second mode), the electronic device receives identification information (eg, the electronic device) of the external electronic device from the external electronic device in a state in which short-range communication is connected with the external electronic device. ID unique information of the external electronic device may be received, and appearance information and function information of the external electronic device may be stored in the memory 330 by matching with the identification information of the external electronic device.

According to an embodiment, in the mode (eg, the second mode), the electronic device receives identification information (eg, the electronic device) of the external electronic device from the external electronic device in a state in which short-range communication is connected with the external electronic device. id information of the external electronic device), the external electronic device's appearance information and the function information of the external electronic device are received, and the external electronic device's appearance information and function information are stored in the memory 330 by matching with the identification information of the external electronic device. there is.

According to an embodiment, the electronic device, in the mode (eg, the second mode), transmits identification information of the external electronic device, appearance information, and function information of the external electronic device to a server (eg, the server of FIG. 2B ( 410)).

11 is a flowchart 1100 for explaining an operation for controlling an external electronic device in the real world in a virtual environment in an electronic device according to various embodiments. Operations capable of controlling an external electronic device in the real world in the virtual environment may include operations 1101 to 1113. According to an embodiment, at least one of operations 1101 to 1113 may be omitted, the order of some operations may be changed, or another operation may be added.

In operation 1101, an electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 210 of FIGS. 2A to 2B , and/or the electronic device 301 of FIG. 3 ) operates an external electronic device in a virtual environment. In a controllable mode (eg, the first mode), a virtual external electronic device may be rendered and displayed based on appearance information and function information of the external electronic device.

According to an embodiment, the electronic device may control an external electronic device in a virtual environment by executing an application for executing virtual reality (VR) or an application for executing virtual reality (VR). Mode (eg, first mode), and external electronic device selected by the user from a list including at least one external electronic device usable in the mode (eg, first mode) and function information are displayed. can be detected.

According to an embodiment, the electronic device, in the mode (eg, the first mode), is an external electronic device selected by a user from among at least one external electronic device stored in a memory (eg, the memory 330 of FIG. 3 ). Appearance information and function information of the device may be detected from the memory.

According to an embodiment, in the electronic device and the mode (eg, the first mode), appearance information and function information of an external electronic device selected by a user from a list including at least one external electronic device stored in the memory are displayed. A request may be made to a server (eg, the server 410 of FIG. 2B ) and received from the server.

According to an embodiment, the electronic device, in the mode (eg, the first mode), external information (eg, 3D coordinate information, texture and shader data) and function information of the external electronic device A virtual external electronic device similar to the external electronic device in the real world may be rendered and displayed by reflecting the rendering technology (eg, graphic processing) constituting the virtual environment.

In operation 1103, the electronic device (eg, the electronic device 101 of FIG. 1, the electronic device 210 of FIGS. 2A to 2B, and/or the electronic device 301 of FIG. 3) receives an input event from the virtual external electronic device. occurrence can be confirmed.

In operation 1103, when the electronic device confirms the occurrence of an input event in the virtual external electronic device, in operation 1105, the electronic device generates input information related to the input event, and converts the input information to the external electronic device (eg, FIGS. 2A to 20). It can be transmitted to the external electronic device 310 of FIG. 2B.

According to an embodiment, the electronic device generates an input event (or touch event) in a display area or a button area of the virtual external electronic device based on a user's gesture in the mode (eg, the first mode) Ex: Dial button input for phone call and/or call button input) is detected, input information including type of input, position of input, and number and duration of input is generated, and a message including the input information is generated. may be transmitted to an external electronic device.

According to an embodiment, in the mode (eg, the first mode), the electronic device transmits a message including the input information to an external electronic device to which short-range communication is connected or to the external electronic device through the server. A message including the input information may be transmitted.

In operation 1107, the electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 210 of FIGS. 2A to 2B , and/or the electronic device 301 of FIG. 3 ) transmits input information to an external device. It is possible to check whether output information is received from the electronic device (eg, the external electronic device 310 of FIGS. 2A and 2B and/or the external electronic device 501 of FIG. 5 ).

In operation 1107, when the electronic device confirms reception of output information from the external electronic device according to the transmission of the input information, based on the output information in operation 1109, the state of the external electronic device changed to the occurrence of an input event corresponds to the state. The state of the virtual external electronic device (eg, the state of the display area or/and the state of sound) may be changed.

According to an embodiment, the electronic device, in the mode (eg, the first mode), according to transmission of a message including the input information and execution of a function corresponding to the input information from the external electronic device When output information related to the changed state of the external electronic device (eg, a screen indicating that a call is being connected for a phone call) is received, the virtual external electronic device corresponds to the changed state of the external electronic device based on the output information can change the state of (eg, displaying a call attempt screen for a phone call on the display area of the virtual electronic device).

According to an embodiment, the electronic device, in the mode (eg, the first mode), receives a message including the output information from an external electronic device to which short-range communication is connected or from the external electronic device through the server. A message including the output information may be received.

In operation 1111, the electronic device (eg, the electronic device 101 of FIG. 1, the electronic device 210 of FIGS. 2A to 2B, and/or the electronic device 301 of FIG. It is possible to check whether output information related to an event generated in the external electronic device 310 of FIGS. 2A to 2B and/or the external electronic device 501 of FIG. 5 is received. In operation 1111, when the electronic device confirms reception of output information related to an event generated in the external electronic device, in operation 1113, the state of the external electronic device changed to the occurrence of an event based on the output information corresponds to the state. The state of the virtual external electronic device (eg, display area state or/and sound state) may be changed.

According to an embodiment, the electronic device, in the mode (eg, the first mode), output information related to an event generated in the external electronic device from the external electronic device (eg, a call connection screen according to a call connection) When a message including is received, the state of the virtual external electronic device is changed to correspond to the state of the external electronic device changed by the occurrence of the event based on the output information (eg, a call is connected to the display area of the external electronic device). call screen according to) can be displayed.

According to an embodiment, the electronic device, in the mode (eg, the first mode), receives a message including the output information from an external electronic device to which short-range communication is connected or from the external electronic device through the server. A message including the output information may be received.

12 is a flowchart 1200 for explaining an operation for controlling an external electronic device in the real world in a virtual environment in a system according to various embodiments of the present disclosure. Operations capable of controlling an external electronic device in the real world in the virtual environment may include operations 1201 to 1223. According to an embodiment, at least one of operations 1201 to 1223 may be omitted, the order of some operations may be changed, or another operation may be added.

In operation 1201, the electronic device 210 (eg, the electronic device 301 of FIG. 3 ) may communicate with the external electronic device 310 (eg, the external electronic device 501 of FIG. 5 ).

According to an embodiment, the electronic device 210 may be connected to the external electronic device 310 through short-range communication through a short-range communication module included in a communication module (eg, the communication module 280 of FIG. 3 ). .

In operation 1203, the electronic device 210 (eg, the electronic device 301 of FIG. 3 ), in a mode capable of controlling the external electronic device in a virtual environment (eg, the first mode), displays information about the external electronic device. And based on the function information, a virtual external electronic device may be rendered and displayed.

According to an embodiment, the electronic device 210 controls an external electronic device in a virtual environment by executing an application for executing virtual reality (VR) or an application for executing virtual reality (VR). switching to a mode (eg, first mode) capable of performing external electronic devices, and displaying external information and function information of an external electronic device selected by a user from among at least one external electronic device usable in the mode (eg, first mode). can be detected.

According to an embodiment, the electronic device 210 stores appearance information and function information of an external electronic device selected by a user from among at least one external electronic device in the mode (eg, the first mode) in a memory (eg, the first mode). It can be detected in the memory 330 of FIG. 3 .

According to an embodiment, the electronic device 310, in the mode (eg, the first mode), provides information on the external appearance of an external electronic device selected by a user from a list including at least one external electronic device stored in the memory. And function information may be requested from a server (eg, the server 410 of FIG. 2B ) and received from the server.

According to an embodiment, the electronic device, in the mode (eg, the first mode), external information (eg, 3D coordinate information, texture and shader data) and function information of the external electronic device A virtual external electronic device similar to the external electronic device in the real world may be rendered and displayed by reflecting the rendering technology (eg, graphic processing) constituting the virtual environment.

In operation 1205, the electronic device 210 (eg, the electronic device 301 of FIG. 3 ) may check whether an input event has occurred in the virtual external electronic device. In operation 1205, if the electronic device 210 confirms the occurrence of an input event in the virtual external electronic device, in operation 1207, it can generate input information related to the input event.

According to an embodiment, the electronic device 210 sends an input event (or touch event) to a display area or button area of the virtual external electronic device based on a user's gesture in the mode (eg, the first mode). When an occurrence of a dial button input and/or a call button input for a phone call is detected, input information including the type of input, the location of the input, and the number and duration of the input may be generated.

In operation 1209, the electronic device 210 (eg, the electronic device 301 of FIG. 3) may transmit input information to the external electronic device 310.

According to an embodiment, the electronic device 210 may transmit a message including the input information to the external electronic device 310 to which short-range communication is connected in the mode (eg, the first mode).

In operation 1211, the external electronic device 310 (eg, the external electronic device 501 of FIG. 5) performs a function corresponding to the input information based on the input information, and according to the execution of the function corresponding to the input information Output information related to the changed state of the external electronic device may be generated.

According to an embodiment, the external electronic device 310 performs a corresponding function in the external electronic device 210 based on the input information received from the electronic device 210, and according to the execution of the function, the A change in the state of the external electronic device (eg, a screen indicating that a call is being connected for a phone call is displayed on a display of the external electronic device) may be confirmed. The external electronic device 310 may generate output information related to the changed state of the external electronic device.

In operation 1213, the external electronic device 310 (eg, the external electronic device 501 of FIG. 5) may transmit output information to the electronic device 210.

According to an embodiment, the external device 310 may transmit a message including the output information to the electronic device 210 to which short-range communication is connected in the mode (eg, the first mode).

In operation 1215, the electronic device (eg, the electronic device 101 of FIG. 1 and/or the electronic device 301 of FIG. 3 ) corresponds to the state of the external electronic device changed due to the occurrence of the input event based on the output information. The state of a virtual external electronic device can be changed.

According to an embodiment, the electronic device, in the mode (eg, the first mode), outputs information related to the state of the external electronic device changed according to the execution of a function corresponding to the input information from the external electronic device. Upon reception, a state of the virtual external electronic device may be changed to correspond to the changed state of the external electronic device based on the output information (eg, displaying a screen indicating that a call is being connected for a phone call).

In operation 1217, the external electronic device 310 (eg, the external electronic device 501 of FIG. 5) may check whether an event has occurred in the external electronic device. In operation 1217, if the external electronic device 310 confirms that an event has occurred, in operation 1219, the external electronic device 310 may generate output information related to the state of the external electronic device changed according to the occurrence of the event.

According to an embodiment, when a state (eg, a display state or/and a sound state) of the external electronic device is changed according to an event generated in the external electronic device, the external electronic device 310 changes the external electronic device 310. The state of the device can be generated as output information.

For example, when an event such as a call reception occurs in the external electronic device 310, the external electronic device 310 receives a call displayed on the display of the external electronic device 310. You can capture the frame buffer for the screen and the sound stream for the ringtone and create it as output information.

In operation 1221, the external electronic device 310 (eg, the external electronic device 501 of FIG. 5) may transmit output information to the electronic device 210.

According to an embodiment, a message including the output information may be transmitted to the electronic device 210 to which short-range communication is connected.

In operation 1223, the electronic device 210 (eg, the external electronic device 501 of FIG. 5 ) determines the state of the virtual external electronic device (eg, the external electronic device 501 of FIG. : Display area state or/and sound state) can be changed.

According to an embodiment, when the electronic device 210 receives a message including output information related to an event generated in the external electronic device from the external electronic device in the mode (eg, the first mode), Based on the output information, the state of the virtual external electronic device may be changed to correspond to the state of the external electronic device changed by the occurrence of the event.

For example, the electronic device 210, in the mode (eg, the first mode), output information received from the external electronic device 310 (eg, a frame buffer for a call reception screen and a bell sound) sound stream), texture mapping is performed on the display area of the virtual external electronic device to display a call reception screen notifying the reception of a call on the display screen of the virtual external electronic device, and the call is received through the speaker of the electronic device. By outputting a ringing sound informing the user, the user can know that a call is being received by an external electronic device in the real world in the virtual environment.

13A to 13B are flowcharts 1300a to 1300b for describing an operation capable of controlling an external electronic device in the real world in a virtual environment in a system according to various embodiments of the present disclosure. Operations capable of controlling an external electronic device in the real world in the virtual environment may include operations 1301a to 1323. According to an embodiment, at least one of operations 1301a to 1323 may be omitted, the order of some operations may be changed, or another operation may be added.

In operation 1301a, the electronic device 210 (eg, the electronic device 301 of FIG. 3 ) connects communication with the server 410, and in operation 1301b, the server 410 connects the external electronic device 310 ( Example: Communication with the external electronic device 501 of FIG. 5 may be connected.

According to an embodiment, if the electronic device 210 succeeds in authentication with the server 410 through an authentication process, it can connect communication.

According to an embodiment, the external electronic device 310 can connect communication when authentication with the server 410 is successful through an authentication process.

In operation 1303, the electronic device 210 (eg, the electronic device 301 of FIG. 3 ), in a mode capable of controlling the external electronic device in a virtual environment (eg, the first mode), external information of the external electronic device And based on the function information, a virtual external electronic device may be rendered and displayed.

According to an embodiment, the electronic device 210 controls an external electronic device in a virtual environment by executing an application for executing virtual reality (VR) or an application for executing virtual reality (VR). switching to a mode (eg, first mode) capable of performing external electronic devices, and displaying external information and function information of an external electronic device selected by a user from among at least one external electronic device usable in the mode (eg, first mode). can be detected.

According to an embodiment, the electronic device 210, in the mode (eg, the first mode), detects a list including at least one external electronic device from a memory (eg, the memory 330 of FIG. 3). and displaying external electronic device information and function information of an external electronic device selected by the user from the list may be detected from the memory.

According to an embodiment, the electronic device 310, in the mode (eg, the first mode), stores a list including at least one external electronic device in a memory (eg, the memory 330 of FIG. 3 ). Detect and display the external electronic device selected by the user from the list and request external information and function information of the external electronic device selected by the user to the server 410) to receive the external electronic device selected by the user from the server 410. there is.

According to an embodiment, the electronic device, in the mode (eg, the first mode), external information (eg, 3D coordinate information, texture and shader data) and function information of the external electronic device A virtual external electronic device similar to the external electronic device in the real world may be rendered and displayed by reflecting the rendering technology (eg, graphic processing) constituting the virtual environment.

In operation 1305, the electronic device 210 (eg, the electronic device 301 of FIG. 3 ) may check whether an input event has occurred in the virtual external electronic device. In operation 1205, if the electronic device 210 confirms the occurrence of an input event in the virtual external electronic device, in operation 1307, it can generate input information related to the input event.

According to an embodiment, the electronic device 210 sends an input event (or touch event) to a display area or button area of the virtual external electronic device based on a user's gesture in the mode (eg, the first mode). When the occurrence of (eg, a dial button input and/or a call button input for a phone call) is detected, input information including the type of input, the location of the input, and the number and duration of the input may be generated.

In operation 1309a, when the electronic device 210 (eg, the electronic device 301 of FIG. 3) transmits input information to the server 410, the server 410 transmits the input information to the external electronic device in operation 1309b. device 310.

According to an embodiment, the electronic device 210 may transmit a message including the input information to the server 410 in the mode (eg, the first mode), and the message includes the external electronic device It may include identification information of.

According to an embodiment, the server 410 may transmit the message to the external electronic device 310 based on a message including the input information and identification information of the external electronic device.

In operation 1311, the external electronic device 310 (eg, the external electronic device 501 of FIG. 5) performs a function corresponding to the input information based on the input information, and according to the execution of the function corresponding to the input information Output information related to the changed state of the external electronic device may be generated.

According to an embodiment, the external electronic device 310 performs a corresponding function in the external electronic device 210 based on input information received from the electronic device 210 through the server 410, , A change in the state of the external electronic device (eg, a screen indicating that a call is being connected for a phone call is displayed on the display of the external electronic device) may be checked according to the function execution. The external electronic device 310 may generate output information related to the changed state of the external electronic device.

In operation 1313a, when the external electronic device 310 (eg, the external electronic device 501 of FIG. 5) transmits output information to the server 410, in operation 1313b, the server 410, Output information may be transmitted to the electronic device 210 .

According to an embodiment, the external electronic device 210 may transmit a message including the output information to the server 410, and the message may include identification information of the electronic device.

According to an embodiment, the server 410 may transmit the message to the electronic device 210 based on a message including the output information and identification information of the electronic device.

In operation 1315, the electronic device (eg, the electronic device 101 of FIG. 1 and/or the electronic device 301 of FIG. 3 ) corresponds to the state of the external electronic device changed by the occurrence of the input event based on the output information. The state of the virtual external electronic device (eg, the state of the display area or/and the state of sound) may be changed.

According to an embodiment, the electronic device, in the mode (eg, the first mode), outputs information related to the state of the external electronic device changed according to the execution of a function corresponding to the input information from the external electronic device. If received, the state of the virtual external electronic device is changed to correspond to the changed state of the external electronic device based on the output information (eg, a screen display indicating that a call is being made for a phone call is displayed on the display area of the virtual electronic device) can do.

In operation 1317, the external electronic device 310 (eg, the external electronic device 501 of FIG. 5) may check whether an event has occurred in the external electronic device. In operation 1317, if the external electronic device 310 confirms that an event has occurred, in operation 1319, the external electronic device 310 may generate output information related to the state of the external electronic device changed according to the occurrence of the event.

According to an embodiment, when a state (eg, a display state or/and a sound state) of the external electronic device is changed according to an event generated in the external electronic device, the external electronic device 310 changes the external electronic device 310. The state of the device can be generated as output information.

For example, when an event such as a call reception occurs in the external electronic device 310, the external electronic device 310 receives a call displayed on the display of the external electronic device 310. You can capture the frame buffer for the screen and the sound stream for the ringtone and create it as output information.

In operation 1321a, the external electronic device 310 (eg, the external electronic device 501 of FIG. 5) transmits output information to the server 410, and in operation 1321b, the server 410 transmits the output information. It can be transmitted to the electronic device 210.

According to an embodiment, the external electronic device 210 may transmit a message including the output information to the server 410, and the message may include identification information of the electronic device.

According to an embodiment, the server 410 may transmit the message to the electronic device 210 based on a message including the output information and identification information of the electronic device.

In operation 1323, the electronic device 210 (eg, the external electronic device 501 of FIG. 5 ) determines the state of the virtual external electronic device (eg, the external electronic device 501 of FIG. : Display area state or/and sound state) can be changed.

According to an embodiment, when the electronic device 210 receives a message including output information related to an event generated in the external electronic device from the external electronic device in the mode (eg, the first mode), Based on the output information, the state of the virtual external electronic device may be changed to correspond to the state of the external electronic device changed by the occurrence of the event.

For example, the electronic device 210, in the mode (eg, the first mode), output information received from the external electronic device 310 (eg, a frame buffer for a call reception screen and a bell sound) sound stream), texture mapping is performed on the display area of the virtual external electronic device to display a call reception screen notifying the reception of a call on the display screen of the virtual external electronic device, and the call is received through the speaker of the electronic device. By outputting a ringing sound informing the user, the user can know that a call is being received by an external electronic device in the real world in the virtual environment.

According to various embodiments, a method for controlling an external electronic device in the real world in a virtual environment in a system may include, in a first mode in which the electronic device controls an external electronic device in a virtual environment of the electronic device, An operation of displaying a virtual external electronic device corresponding to the external electronic device based on appearance information and function information of the external electronic device, and when an input event is detected by the electronic device, the external electronic device Transmitting input information related to the input event to an electronic device, performing a function corresponding to the input information when the external electronic device receives the input information from the electronic device, and performing a function corresponding to the input information An operation of transmitting output information related to a state of an external electronic device changed according to the execution of a function to the electronic device, and when the output information is received from the external electronic device by the electronic device, the virtual electronic device is configured based on the output information. An operation of changing the state of the external electronic device to correspond to the changed state of the external electronic device.

According to various embodiments, in a second mode in which the electronic device can register an external electronic device in a virtual environment of the electronic device, the external electronic device is recognized, and based on the recognition of the external electronic device, the external electronic device is recognized. An operation of generating outer appearance information of an external electronic device, an operation of displaying, by the electronic device, a virtual external electronic device generated based on the outer appearance information and a function setting menu capable of setting functions of the virtual electronic device, Generating, by a device, information on a type of at least one function selected through the function setting menu and an area where a component for performing the at least one function in the virtual external electronic device is located as the function information; and An operation of storing, by the electronic device, the appearance information and the function information to be matched with the external electronic device.

According to various embodiments, at least one function included in the function setting menu may be pre-stored in the electronic device, received from the external electronic device, or set to be received from a server.

According to various embodiments of the present disclosure, when output information related to an event generated in the external electronic device is received from the external electronic device in the first mode by the electronic device, the event is changed to the occurrence of the event based on the output information. An operation of changing a state of the virtual external electronic device to correspond to a state of the external electronic device may be further included.

According to various embodiments of the present disclosure, when an event occurs in the external electronic device and the state of the external electronic device is changed, the output information related to the changed state of the external electronic device is generated, and the electronic device An operation of transmitting to may be further included.

According to various embodiments, if the electronic device is in a state in which short-range communication is connected with the external electronic device, an operation of transmitting and receiving information with the external electronic device based on the short-range communication connection may be further included.

According to various embodiments, if the electronic device is in a state in which short-range communication is not connected with the external electronic device, an operation of transmitting and receiving information of the external electronic device through a server may be further included.

According to various embodiments of the present disclosure, a method for controlling an external electronic device in the real world in a virtual environment in an electronic device may include external electronic device information and functions in a first mode capable of controlling an external electronic device in a virtual environment. An operation of displaying a virtual external electronic device based on information. When an input event is detected in the virtual external electronic device, input information related to the input event is transmitted to the external electronic device, and the input information is transmitted from the external electronic device. displaying the state of the virtual external electronic device in correspondence with the changed state of the external electronic device based on the output information when output information related to the state of the external electronic device changed according to the execution of the function corresponding to and, upon receiving output information related to an event generated in the external electronic device from the external electronic device, changing a state of the virtual external electronic device to correspond to a state of the external electronic device changed based on the output information. can include

According to various embodiments, in a second mode capable of registering an external electronic device in a virtual environment, the external electronic device is recognized through the camera module, and the exterior of the external electronic device is changed based on the recognition of the external electronic device. An operation of generating information, an operation of displaying a virtual external electronic device generated based on the appearance information and a menu for setting functions of the virtual electronic device, a type of at least one function selected through the menu, and the virtual external electronic device. The method may further include generating a location of a function to be selected to perform the at least one function in the electronic device as the function information, and storing the appearance information and the function information to match the external electronic device. there is.

According to various embodiments of the present disclosure, when the electronic device is connected to the external electronic device through short-range communication, transmitting/receiving information with the external electronic device based on the short-range communication connection, and the electronic device communicating with the external electronic device If short-range communication is not connected, an operation of transmitting/receiving information of the external electronic device through a server may be further included.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document are a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, the electronic device 101 or the electronic device 301). ) may be implemented as software (eg, program 140) including one or more instructions stored in . For example, a processor (eg, the processor 320 ) of a device (eg, the electronic device 301 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

Claims (20)

In a system capable of controlling an external electronic device in the real world in a virtual environment,
In a first mode capable of controlling an external electronic device in a virtual environment, a virtual external electronic device corresponding to the external electronic device is displayed based on appearance information and function information of the external electronic device, and an input is input from the virtual external electronic device. When the occurrence of an event is detected, input information related to the input event is transmitted to the external electronic device, and output information related to the state of the external electronic device changed according to the execution of a function corresponding to the input information from the external electronic device. an electronic device that changes a state of the virtual external electronic device to correspond to the changed state of the external electronic device based on the output information when receiving the ; and
When receiving the input information from the electronic device, performing a function corresponding to the input information and transmitting output information related to a state of the external electronic device changed according to the execution of the function corresponding to the input information to the electronic device A system that includes external electronics.
The method of claim 1, wherein the electronic device,
In a second mode for registering an external electronic device in a virtual environment, recognizing the external electronic device and generating external information of the external electronic device based on the recognition of the external electronic device;
Displaying a virtual external electronic device generated based on the appearance information and a function setting menu for setting functions of the virtual electronic device;
Generating, as the function information, a type of at least one function selected through the function setting menu and information on an area where a component for performing the at least one function in the virtual external electronic device is located;
A system configured to store the appearance information and the function information to be matched with the external electronic device.
According to claim 1,
At least one function included in the function setting menu is set to be pre-stored in the electronic device, received from the external electronic device, or received from a server.
The method of claim 1, wherein the electronic device,
In the first mode, when output information related to an event generated in the external electronic device is received from the external electronic device, the virtual external device corresponds to a state of the external electronic device that has been changed due to the occurrence of the event based on the output information. A system set up to change the state of an electronic device.
The method of claim 4, wherein the external electronic device,
The system configured to generate and transmit the output information related to the changed state of the external electronic device to the electronic device when an event occurs in the external electronic device and the state of the external electronic device is changed.
According to claim 1,
The system configured to transmit and receive information with the external electronic device based on the short-range communication connection when the electronic device is in a short-range communication connection state with the external electronic device.
According to claim 1,
The system configured to transmit and receive information of the external electronic device through a server when the electronic device is not connected to the external electronic device through short-range communication.
An electronic device capable of controlling an external electronic device in the real world in a virtual environment,
camera module; and
It includes a processor, the processor comprising:
In a first mode for controlling an external electronic device in a virtual environment, a virtual external electronic device is displayed based on appearance information and function information of the external electronic device;
When the virtual external electronic device detects the occurrence of an input event, it transmits input information related to the input event to the external electronic device, and the external electronic device changes according to the execution of a function corresponding to the input information from the external electronic device. When output information related to the state of is received, based on the output information, the state of the virtual external electronic device is displayed in correspondence with the changed state of the external electronic device;
When receiving output information related to an event generated in the external electronic device from the external electronic device, the electronic device configured to change the state of the virtual external electronic device to correspond to the state of the external electronic device changed based on the output information. .
The method of claim 8, wherein the processor,
In a second mode in which an external electronic device can be registered in a virtual environment, the external electronic device is recognized through the camera module, and appearance information of the external electronic device is generated based on the recognition of the external electronic device;
Displaying a virtual external electronic device generated based on the appearance information and a menu for setting functions of the virtual electronic device;
generating a type of at least one function selected through the menu and a position of a function to be selected to perform the at least one function in the virtual external electronic device as the function information;
An electronic device configured to store the appearance information and the function information to be matched with the external electronic device.
The method of claim 8, wherein the processor,
When the electronic device is in a short-range communication connection state with the external electronic device, transmits and receives information with the external electronic device based on the short-range communication connection;
An electronic device configured to transmit/receive information of the external electronic device through a server when the electronic device is not connected to the external electronic device through short-range communication.
A method for controlling an external electronic device in the real world in a virtual environment in a system,
In a first mode in which the electronic device can control an external electronic device in a virtual environment of the electronic device, a virtual external electronic device corresponding to the external electronic device is displayed based on external information and function information of the external electronic device. action;
transmitting, by the electronic device, input information related to the input event to the external electronic device when the occurrence of an input event is detected in the virtual external electronic device;
When the input information is received by the external electronic device, a function corresponding to the input information is performed, and output information related to the state of the external electronic device changed according to the execution of the function corresponding to the input information transmitting to the electronic device; and
and changing, by the electronic device, a state of the virtual external electronic device to correspond to the changed state of the external electronic device based on the output information when output information is received from the external electronic device.
According to claim 11,
In a second mode in which the electronic device can register an external electronic device in a virtual environment of the electronic device, the external electronic device is recognized, and external information of the external electronic device is recognized based on the recognition of the external electronic device. the operation of generating;
displaying, by the electronic device, a virtual external electronic device generated based on the appearance information and a function setting menu capable of setting functions of the virtual electronic device;
Generating, by the electronic device, the type of at least one function selected through the function setting menu and information on a region where a component for performing the at least one function in the virtual external electronic device is located as the function information. ; and
and storing, by the electronic device, the appearance information and the function information to be matched with the external electronic device.
According to claim 11,
At least one function included in the function setting menu is pre-stored in the electronic device, received from the external electronic device, or set to be received from a server.
According to claim 11,
When output information related to an event generated in the external electronic device is received by the electronic device in the first mode from the external electronic device, the state of the external electronic device is changed to the occurrence of the event based on the output information. and changing a state of the virtual external electronic device to correspond to the virtual external electronic device.
According to claim 14,
When an event occurs in the external electronic device and the state of the external electronic device is changed by the external electronic device, the output information related to the changed state of the external electronic device is generated and transmitted to the electronic device. How to include.
According to claim 11,
The method further comprising transmitting/receiving information with the external electronic device based on the short-range communication connection when short-range communication is connected with the external electronic device by the electronic device.
According to claim 11,
The method further comprising transmitting/receiving information of the external electronic device through a server when short-range communication is not connected with the external electronic device by the electronic device.
A method for controlling an external electronic device in the real world in a virtual environment in an electronic device,
displaying a virtual external electronic device based on appearance information and function information of the external electronic device in a first mode capable of controlling an external electronic device in a virtual environment;
When the virtual external electronic device detects the occurrence of an input event, it transmits input information related to the input event to the external electronic device, and the external electronic device changes according to the execution of a function corresponding to the input information from the external electronic device. displaying a state of the virtual external electronic device corresponding to the changed state of the external electronic device based on the output information when output information related to a state of is received; and
and changing a state of the virtual external electronic device to correspond to a state of the external electronic device changed based on the output information when output information related to an event generated in the external electronic device is received from the external electronic device. How to.
According to claim 18,
recognizing the external electronic device through the camera module in a second mode capable of registering the external electronic device in a virtual environment, and generating appearance information of the external electronic device based on the recognition of the external electronic device;
displaying a virtual external electronic device generated based on the appearance information and a menu for setting functions of the virtual electronic device;
generating a type of at least one function selected through the menu and a position of a function to be selected to perform the at least one function in the virtual external electronic device as the function information; and
The method further comprising storing the appearance information and the function information to be matched with the external electronic device.
According to claim 18,
transmitting/receiving information with the external electronic device based on the short-range communication connection when the electronic device is in a short-range communication connection state with the external electronic device; and
The method further comprising transmitting/receiving information of the external electronic device through a server when the electronic device is in a state in which short-range communication is not connected to the external electronic device.

Images