KR20240029482A - A head mounted electronic device that converts a screen of electronic device into extended reality and an electronic device connected thereto - Google Patents

Abstract

A head-worn electronic device that converts the screen of the electronic device into an extended reality environment and an electronic device connected thereto are disclosed. The disclosed electronic device can be connected to a head-worn electronic device. Electronic devices may include memory. The electronic device may include a display. The electronic device may include a processor that receives from a memory a plurality of objects related to the execution screen of the application that the electronic device is executing, to be displayed on the execution screen of the application that is stored in the memory. The display may display an execution screen of the application. The execution screen may include multiple objects. A plurality of objects may display content related to an application. The electronic device may include a processor that converts the plurality of objects into a plurality of virtual objects for display in a visual image output by a display of the head-worn electronic device. The electronic device may include a processor that checks information about the theme corresponding to the color of the execution screen of the application and information about the visual image output by the display of the head-worn electronic device. The electronic device may include a processor that controls at least one of brightness and saturation of a plurality of virtual objects based on at least one of information about a theme and information about a visual image.

Description

A head-worn electronic device that converts the screen of the electronic device into an extended reality environment and an electronic device connected thereto

Embodiments of the present disclosure relate to a head-worn electronic device that transforms the screen of the electronic device into an extended reality environment. Embodiments of the present disclosure relate to head-worn electronic devices and connected electronic devices.

Head-worn electronic devices are being developed that provide users with experiences of augmented reality, virtual reality, mixed reality, and/or eXtended reality. The above-mentioned background technology is possessed or acquired in the process of deriving this disclosure and cannot necessarily be said to be known technology disclosed to the general public before the application of this disclosure.

According to one embodiment, the electronic device may be connected to a head-worn electronic device. Electronic devices may include memory. The electronic device may include a display. The electronic device may include a processor that receives from a memory a plurality of objects related to the execution screen of the application that the electronic device is executing, to be displayed on the execution screen of the application that is stored in the memory. The display may display an execution screen of the application. The execution screen may include multiple objects. A plurality of objects may display content related to an application. The electronic device may include a processor that converts the plurality of objects into a plurality of virtual objects for display in a visual image output by a display of the head-worn electronic device. The electronic device may include a processor that checks information about the theme corresponding to the color of the execution screen of the application and information about the visual image output by the display of the head-worn electronic device. The electronic device may include a processor that controls at least one of brightness and saturation of a plurality of virtual objects based on at least one of information about a theme and information about a visual image.

According to one embodiment, a method of operating an electronic device receives information including a plurality of objects related to the execution screen of an application stored in the memory to be displayed on the execution screen of the application being executed by the electronic device from the memory of the electronic device. It may include actions such as: The display of the electronic device may display an execution screen of an application. The execution screen may include multiple objects. A plurality of objects may display content related to an application. A method of operating an electronic device may include converting a plurality of objects included in information into a plurality of virtual objects for display in a visual image output by a display of the head-worn electronic device. The method of operating the electronic device may include checking information about a theme corresponding to the color of the execution screen of the application and information about a visual image output by the display of the head-worn electronic device. A method of operating an electronic device may include controlling at least one of brightness and saturation of a plurality of virtual objects based on at least one of information about a theme and information about a visual image.

According to one embodiment, the head-worn electronic device may include a camera. The head worn electronic device may include a display. The head-worn electronic device may include a processor that controls a camera to capture the user's surrounding environment. The head-worn electronic device may include a processor that controls the display to display a first visual image that outputs the user's surrounding environment captured by a camera or a second visual image that outputs a virtual environment. The processor may receive information including a plurality of objects related to the execution screen of the application being executed from the electronic device. The display of the electronic device may display an execution screen of an application. The execution screen may include multiple objects. A plurality of objects may display content related to the application. The processor may convert a plurality of objects included in the information into a plurality of virtual objects to be displayed on a visual image output by the display of the head-worn electronic device. The processor can check information about the theme corresponding to the color of the application execution screen and information about the visual image output by the display of the head-worn electronic device. The processor may control at least one of the brightness and saturation of the plurality of virtual objects based on at least one of information about the theme and information about the visual image.

According to one embodiment, an operation method performed by a head-worn electronic device may include receiving information including a plurality of objects related to an execution screen of a running application from the electronic device. The display of the electronic device may display an execution screen of an application. The execution screen may include multiple objects. A plurality of objects may display content related to the application. The operation method performed by the head-worn electronic device may include converting a plurality of objects included in information into a plurality of virtual objects for display in a visual image output by the display of the head-worn electronic device. The operation method performed by the head-worn electronic device may include checking information about a theme corresponding to the color of the execution screen of an application and information about a visual image output by the display of the head-worn electronic device. An operation method performed by a head-worn electronic device may control at least one of brightness and saturation of a plurality of virtual objects based on at least one of information about a theme and information about a visual image.

A non-transitory computer-readable storage medium according to one embodiment may store one or more programs. One or more programs may include an operation of receiving information including a plurality of objects related to the execution screen of an application stored in the memory to be displayed on the execution screen of the application running on the electronic device from the memory. The display of the electronic device may display an execution screen of an application. The execution screen may include multiple objects. A plurality of objects may display content related to an application. One or more programs may include converting a plurality of objects included in information into a plurality of virtual objects for display in a visual image output by a display of the head-worn electronic device. One or more programs may include an operation of checking information about a theme corresponding to the color of the execution screen of the application and information about a visual image output by the display of the head-worn electronic device. One or more programs may include an operation of controlling at least one of brightness and saturation of a plurality of virtual objects based on at least one of information about a theme and information about a visual image.

A non-transitory computer-readable storage medium according to one embodiment may store one or more programs. One or more programs may include an operation of receiving information including a plurality of objects related to an execution screen of an application being executed from an electronic device. The display of the electronic device may display an execution screen of an application. The execution screen may include multiple objects. A plurality of objects may display content related to an application. One or more programs may perform an operation to convert a plurality of objects included in information into a plurality of virtual objects to be displayed in a visual image output by a display of the head-worn electronic device. One or more programs may perform an operation to check information about a theme corresponding to the color of the execution screen of the application and information about a visual image output by the display of the head-worn electronic device. One or more programs may perform an operation to control at least one of brightness and saturation of a plurality of virtual objects based on at least one of information about a theme and information about a visual image.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
Figure 2 is a front perspective view of a head-worn electronic device according to an embodiment of the present disclosure.
Figure 3 is a rear perspective view of a head-worn electronic device according to an embodiment of the present disclosure.
FIG. 4 is a diagram illustrating a method of converting the screen of an electronic device to an XR environment according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating an execution screen of an application according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating conversion to an XR environment according to a theme corresponding to the color of the execution screen of an application according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating a method of converting each screen of an electronic device into a different visual image according to an embodiment of the present disclosure.
Figure 8 is a diagram for explaining a method of converting the size of a font according to an embodiment of the present disclosure.
Figure 9 is a diagram for explaining an object identified according to depth according to an embodiment of the present disclosure.
FIG. 10 is a diagram illustrating objects identified according to depth by layer according to an embodiment of the present disclosure.
FIG. 11 is a diagram illustrating content identified according to depth according to an exemplary embodiment of the present disclosure.
FIG. 12 is a diagram for explaining a virtual object according to a change in a visual image according to an embodiment of the present disclosure.
FIG. 13 is a diagram for explaining screen expansion in an XR environment according to an embodiment of the present disclosure.
Figure 14 is a diagram for explaining the display of a pop-up object according to an embodiment of the present disclosure.
FIG. 15 is a flowchart for explaining a method of operating an electronic device according to an embodiment of the present disclosure.
FIG. 16 is a flowchart for explaining a method of operating a head-worn electronic device according to an embodiment of the present disclosure.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a secondary processor 123, the secondary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support high frequency bands (eg, mmWave bands), for example, to achieve high data rates. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected to the plurality of antennas by, for example, the communication module 190. can be selected. Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197. According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

In one embodiment, the external electronic device 102 may render content data executed in an application and then transmit it to the electronic device 101. The electronic device 101 that received the data may output the content data to the display module 160. If the electronic device 101 detects user movement through an IMU sensor or the like, the processor 120 of the electronic device 101 corrects the rendering data received from the external electronic device 102 based on the movement information. It can be output to the display module 106. Alternatively, the motion information may be transmitted to the external electronic device 102 and rendering may be requested so that the screen data is updated accordingly. According to various embodiments, the external electronic device 102 may be various types of devices, such as a smartphone or a case device that can store and charge the electronic device 101.

Figure 2 is a front perspective view of a head-worn electronic device according to one embodiment. Figure 3 is a rear perspective view of a head-worn electronic device according to an embodiment.

Referring to FIGS. 2 and 3 , a head-worn electronic device 201 (eg, electronic device 101 of FIG. 1 ) may be worn on a part of the user's body and provide a user interface. For example, the head-worn electronic device 201 may provide a user with augmented reality (AR), virtual reality (VR), mixed reality (MR), and/or extended reality. XR) experience.

According to one embodiment, the head-worn electronic device 201 may include a housing 210. Housing 210 may be configured to accommodate at least one component.

According to one embodiment, the head-worn electronic device 201 may include a lens structure 220. The lens structure 220 may include a plurality of lenses 230 configured to adjust the focus of the image provided to the user. For example, the plurality of lenses 230 may be configured to adjust the focus of the image output by the display 240. The plurality of lenses 230 may be disposed at a position corresponding to the position of the display 240. The plurality of lenses 230 may include, for example, a Fresnel lens, a pancake lens, a multi-channel lens, and/or any other suitable lens. The lens structure 220 may be disposed on the rear side of the housing 210.

According to one embodiment, the head-worn electronic device 201 may include a display 240 (eg, display module 160 of FIG. 1). The display 240 may be configured to provide an image (eg, a virtual environment) to the user. For example, the display 240 may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic It may include an organic light emitting diode (OLED), and/or a micro light emitting diode (micro LED). According to one embodiment, when the display 240 is one of LCD, DMD, and LCoS, the display 240 may include a light source (not shown) configured to transmit an optical signal to an area where an image is output. According to one embodiment, when the display 240 is an OLED or micro LED, the display 240 can provide an image to the user by generating an optical signal on its own. According to one embodiment, the display 240 may provide a different experience to the user depending on whether the first camera 260 is operating. For example, when the first camera 260 is operating, the display 240 may display the external environment captured by the first camera 260 to provide an AR or visual see-through (VST) experience to the user. there is. For example, when the first camera 260 is not operating, the display 240 may display a virtual environment to provide a VR experience to the user.

In one embodiment, the head-worn electronic device 201 may include a sensor 250 (e.g., sensor module 176 of FIG. 1). The sensor 250 may be configured to detect the depth of the subject. The sensor 250 may be configured to transmit a signal toward and/or receive a signal from the subject. For example, the transmitted signal may include near-infrared light, ultrasound, and/or laser. The sensor 250 may be configured to measure the time of flight (ToF) of a signal to measure the distance between the head-worn electronic device 201 and the subject.

In one embodiment, the head-worn electronic device 201 may include a plurality of first cameras 260 (eg, the camera module 180 of FIG. 1). The plurality of first cameras 260 may be configured to acquire images from a subject. A plurality of first cameras 260 may be disposed on the front of the housing 210. A plurality of first cameras 260 may be disposed on both sides of the sensor 250, respectively. The plurality of first cameras 260 may include an image stabilizer actuator (not shown) and/or an autofocus actuator (not shown). For example, the plurality of first cameras 260 may include at least one of a camera configured to acquire a color image, a global shutter camera, or a rolling shutter camera, or a combination thereof. According to one embodiment, the first camera 206 can selectively photograph the user's surrounding environment according to the user's command. For example, when a closed mode command is received from a user, the first camera 206 may be turned off. For example, when an open mode command is received from a user, the first camera 206 may be turned on.

In one embodiment, the head-worn electronic device 201 may include a plurality of second cameras 270 (eg, the camera module 180 of FIG. 1). The plurality of second cameras 270 may be configured to recognize a subject. The plurality of second cameras 270 may be configured to detect and/or track an object (eg, a human head or hand) or space with 3 degrees of freedom or 6 degrees of freedom. For example, the plurality of second cameras 270 may include global shutter cameras. The plurality of second cameras 270 may be configured to perform simultaneous localization and mapping (SLAM) using depth information of the subject. The plurality of second cameras 270 may be configured to recognize the subject's gesture. In one embodiment, a plurality of second cameras 270 may be disposed on the front of the housing 210. In one embodiment, a plurality of second cameras 270 may be respectively disposed in corner areas of the front of the housing 210.

In one embodiment, the head-worn electronic device 201 may include a plurality of third cameras 280 (eg, the camera module 180 of FIG. 1). The plurality of third cameras 280 may be configured to recognize the user's face. For example, the plurality of third cameras 280 may be configured to detect and track the user's facial expression.

In one embodiment, although not shown, the head-worn electronic device 201 may include a processor (eg, processor 120 of FIG. 1). The processor may be included inside the housing 210. Alternatively, the processor may be included in a temple (not shown) that is connected to the head-worn electronic device 201 and secures it in close contact with the head of the user of the head-worn electronic device 201. The processor may be electrically and/or physically connected to the display 240, sensor 250, first camera 260, second camera 270, and third camera 280. The processor may control the display 240, sensor 250, first camera 260, second camera 270, and third camera 280. Processors include, for example, Central Processing Units, microprocessors, Graphics Processing Units, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), and Digital Signal Processing Devices (DSPDs). ), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but are not limited thereto.

In an embodiment not shown, the head-worn electronic device 201 may include a microphone (e.g., input module 150 of FIG. 1), a speaker (e.g., sound output module 155 of FIG. 1), and a battery (e.g., Suitable for batteries 189 in FIG. 1), antennas (e.g., antenna module 197 in FIG. 1), sensors (e.g., sensor module 176 in FIG. 1), and/or other head-worn electronic devices 201. Can contain arbitrary components.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

FIG. 4 is a diagram 400 for explaining a method of converting the screen of an electronic device to an XR environment according to an embodiment of the present disclosure.

Referring to FIG. 4 , an electronic device 401 (eg, electronic device 102 or 104 of FIG. 1 ) and a visual image 430 are shown.

The electronic device 401 may include a display 402. The display 402 can display a screen. The display 402 may display an execution screen of an application that the electronic device 401 is executing. Accordingly, the electronic device 401 may be a device that includes a display 402 controlled by the processor of the electronic device 401 to display an execution screen of an application.

For example, the electronic device 401 may include various computing devices such as a mobile phone, smart phone, tablet, e-book device, laptop, personal computer, desktop, workstation, or server, smart watch, smart glasses, or Head-Mounted Display (HMD). ), various home appliances such as smart speakers, smart TVs, or smart refrigerators, smart cars, smart kiosks, Internet of Things (IoT) devices, Walking Assist Devices (WADs), drones, or robots. there is.

The execution screen of the application may include a plurality of objects 410 that can be converted to be displayed in the visual image 430. A plurality of objects may be related to the execution screen of an application. A plurality of objects may display content related to an application.

According to one embodiment, the plurality of objects 410 are related to the execution screen of an application and may be stored in the memory of the electronic device 401. The processor of the electronic device 401 may receive information including a plurality of objects 410 related to the execution screen of the application from the memory. The processor of the electronic device 401 may convert a plurality of objects 410 included in the received information into a plurality of virtual objects 420 to display them in the visual image 430.

According to one embodiment, the plurality of objects 410 are related to the execution screen of an application and may be included in the execution screen of the application. The processor of the electronic device 401 may separate a plurality of objects 410 from the execution screen of the application. The processor of the electronic device 401 may convert the separated plurality of objects 410 into a plurality of virtual objects 420 to display them in the visual image 430 .

According to one embodiment, the plurality of objects 410 may include a first object 411 displayed at the top of the screen. The plurality of objects 410 may include a second object 412 displayed in the middle of the screen. The plurality of objects 410 may include a third object 413 displayed at the bottom of the screen. For example, the application execution screen displayed by the electronic device 401 may be the execution screen of a call application. When running a call application, the display displays a first object at the top that can be switched to a call record search or option screen, a second object that displays call records in the middle, and a third object that can be switched to a screen such as a numeric keypad at the bottom. Objects can be displayed. The application execution screen displayed on the display 402 is only an example to explain the invention and is not limited thereto.

According to one embodiment, an object displayed on the electronic device 401 may include one or more detailed objects related to the object. According to one embodiment, a virtual object displayed on a head-worn electronic device may include one or more detailed virtual objects related to the virtual object. For example, if the object is related to yesterday's call history, the object may include a first detail object displaying call history yesterday morning and a second detail object displaying call history yesterday afternoon.

A head-worn electronic device (e.g., the electronic device 101 of FIG. 1 and the head-worn electronic device 201 of FIG. 2) may output a visual image 430 according to a user's command. When a user wears temples (not shown) that can be connected to a head-worn electronic device and places his/her eyes in close contact with a lens structure (e.g., lens structure 220 in FIG. 2), the head-worn electronic device A visual image 430 may be provided.

According to one embodiment, the visual image 430 is the user's surrounding environment captured by a camera included in the head-worn electronic device (e.g., the camera module 180 in FIG. 1 and the first camera 260 in FIG. 2). It may be a first visual image output. For example, the first visual image may be AR or VST. The AR or VST environment may be when the first camera of FIG. 2 is turned on. VST can be an experience that shows the external environment in the direction the user is facing when the user leaves a specific designated area in a VR environment. For example, when the user leaves a specific area pre-designated for safety reasons in the VR environment, the external environment captured through the first camera of FIG. 2 may be displayed on the display for safety reasons.

According to one embodiment, the visual image 430 may be a second visual image that outputs a virtual environment when the camera is turned off. For example, the second visual image may be VR.

The head-worn electronic device may be connected to the electronic device 401 wired or wirelessly. At this time, a processor included in the head-worn electronic device (e.g., processor 120 in FIG. 1) may determine whether the head-worn electronic device and the electronic device 401 are connected. According to one embodiment, the processor of the electronic device 401 may determine whether the head-worn electronic device and the electronic device 401 are connected.

When the head-worn electronic device is connected to the electronic device 401 by wire or wirelessly, the processor of the head-worn electronic device displays a plurality of objects 410 that can be converted for display in the visual image 430. It may be determined whether the application is running on the electronic device 401. According to one embodiment, the processor of the electronic device 401 determines whether an application displaying a plurality of objects 410 that can be converted to be displayed in the visual image 430 is running on the electronic device 401. can do.

If the plurality of objects 410 cannot be converted to be displayed in the visual image 430, the processor of the head-worn electronic device may change the display of the head-worn electronic device (e.g., the display module 160 of FIG. 1 ) and the display 240 of FIG. 2) can simply mirror the screen of the electronic device 401. According to one embodiment, if the plurality of objects 410 cannot be converted to be displayed in the visual image 430, the processor of the electronic device 401 may cause the display of the head-worn electronic device to simply display the electronic device 401. ) screen can be mirrored.

The processor of the head-worn electronic device may convert a plurality of objects 410 included in information received from the electronic device 401 into a plurality of virtual objects 420 to display them in the visual image 430. . According to one embodiment, the processor of the electronic device 401 uses a plurality of virtual objects to display the plurality of objects 410 included in the information received from the memory of the electronic device 401 on the visual image 430. It can be converted to (420). For example, the first object 411 may be converted into a first virtual object 421. The second object 412 may be converted into a second virtual object 422. The third object 413 may be converted into a third virtual object 423.

According to one embodiment, the plurality of virtual objects 420 may be displayed in any area desired by the user within the visual image 430. For example, when a user selects next to a specific object displayed in the visual image 430, a plurality of virtual objects 420 may be displayed in the selected area.

According to one embodiment, the processor of the head-worn electronic device may arrange a plurality of virtual objects 420 in the visual image 430 based on the arrangement status of the plurality of objects 410. A plurality of virtual objects 420 may be arranged in the visual image 430 according to the arrangement order of the plurality of objects 410. According to one embodiment, the processor of the electronic device may arrange a plurality of virtual objects 420 in the visual image 430 based on the arrangement status of the plurality of objects 410. A plurality of virtual objects 420 may be arranged in the visual image 430 according to the arrangement order of the plurality of objects 410. For example, the first virtual object 421, the second virtual object 422, and the third virtual object 423 each correspond to the first object 411, the second object 412, and the third object 413. ) can be placed in the visual image 430 according to the arrangement order.

According to one embodiment, the plurality of virtual objects 420 may be arranged in the visual image 430 in the same arrangement as the arrangement of the plurality of objects 410 having the same name. For example, referring to FIG. 4 , the first object 411, the second object 412, and the third object 413 may be arranged in order from top to bottom. Since the first object 411 and the first virtual object 421 have the same app bar (APP bar) name, the first virtual object 421 may be arranged in the same arrangement as the first object 411.

According to one embodiment, the plurality of virtual objects 420 can perform all operations that the plurality of objects 410 can perform. For example, when the user selects the search icon on the first virtual object 421, the same event may occur as when the user selects the search icon on the first object 411 on the electronic device 401. According to one embodiment, the second object 412 may be a scrollable element. From the top to the bottom of the second object 412 may be a scroll area 414. For example, when the user scrolls up the screen on which the second object 412 is displayed, information that could not be displayed due to the size of the screen may be scrolled upward and displayed. Accordingly, the second virtual object 422 having the same name as the second object 412 may also be a scrollable object.

The plurality of virtual objects 420 are displayed floating in the visual image 430 so that the user can feel the plurality of virtual objects 420 floating in space. Additionally, the user can experience a consistent UX (User Experience) by displaying a plurality of virtual objects 420 through an interface similar to the user interface of the electronic device 401. Additionally, regardless of the type of electronic device 401 to which the head-worn electronic device is connected, if the electronic device 401 can execute an application that displays an object that can be converted to be displayed in the visual image 430, the user can be provided with a consistent connection regardless of the type of terminal.

In this specification, the camera and display of the head-worn electronic device may be controlled by the processor of the head-worn electronic device or the processor of the electronic device 401. Additionally, in this specification, the plurality of virtual objects 420 may be converted and controlled by the processor of the head-worn electronic device or the processor of the electronic device. Operations described herein may be performed or controlled by a processor of the head-worn electronic device or a processor of the electronic device. And, even if not specifically mentioned, the operations described in this specification may be performed or controlled by the processor of the head-worn electronic device or the processor of the electronic device.

Below, we will describe a user interface that can be converted to display in a visual image.

FIG. 5 is a diagram 500 illustrating an execution screen of an application according to an embodiment of the present disclosure.

Referring to FIG. 5, an electronic device 501 (e.g., the electronic device 102 or 104 of FIG. 1 and the electronic device 104 of FIG. 4) displays an execution screen of an application including a plurality of convertible objects 410. 401)) is shown.

The electronic device 501 may include a display 502 (eg, display 402 in FIG. 4). The display 502 may display an execution screen of an application including a plurality of objects 410 that can be converted to be used in XR environments such as VR, AR, and VST environments.

According to one embodiment, when the electronic device 501 displays a plurality of objects 410 that cannot be converted for use in an XR environment, the head-worn electronic device (e.g., the electronic device 101 of FIG. 1 and The head-worn electronic device 201 of FIG. 2 can simply mirror and display the screen of the electronic device 501. According to one embodiment, when the electronic device 501 displays a plurality of objects 410 that can be converted for use in an XR environment, the head-worn electronic device displays a plurality of objects 410 included in the user interface. may be converted into a plurality of virtual objects (e.g., a plurality of virtual objects 420 in FIG. 4) provided to the user in an XR environment and displayed. Therefore, users can experience a consistent UX (User Experience).

The execution screen of the application may include a background object 503 forming the background and a plurality of objects 410 arranged on the background object 503. For example, the plurality of objects 410 may include a first object 411, a second object 412, and a third object 413.

According to one embodiment, the execution screen of the application may display a plurality of objects 410 related to the execution screen of the application, and the plurality of objects 410 may display content related to the application.

The names of the plurality of objects 410 that enable communication between the user and the electronic device 501 may be predetermined. Each of the plurality of objects 410 may be assigned a different name. An object can group and display content for applications related to the object. For example, when the first object 411 is related to an application that displays to-dos, the first object 411 may display content related to the application that displays to-dos.

According to one embodiment, each of the plurality of objects 410 may display content grouped by different criteria. According to one embodiment, contents included in the same object may be contents grouped based on the same criteria. Accordingly, contents belonging to different objects may be contents grouped by different criteria.

According to one embodiment, contents included in the same object may be grouped and displayed again into a plurality of detailed objects. Contents included in the same object may be grouped into a plurality of detailed objects based on lower-level criteria. For example, content included in call history, which is the same object, may be grouped and displayed into a plurality of detailed objects based on date criteria, which are lower criteria.

Each of the plurality of virtual objects (e.g., the plurality of virtual objects 420 of FIG. 4) is converted to display the plurality of objects 410 in a visual image (e.g., the visual image 430 of FIG. 4). Since it is a virtual object, the description of the plurality of objects 410 described above can all be applied.

According to one embodiment, names of a plurality of virtual objects that enable communication between the user and the electronic device 501 may be predetermined. Each of the plurality of virtual objects may be assigned a different name.

According to one embodiment, each of the plurality of virtual objects may display contents grouped by different criteria. According to one embodiment, contents included in the same virtual object may be contents grouped by the same criteria. Accordingly, contents belonging to different virtual objects may be contents grouped according to different criteria.

According to one embodiment, contents included in the same virtual object may be grouped and displayed again into a plurality of detailed virtual objects. Contents included in the same virtual object may be grouped into a plurality of detailed objects based on lower-level criteria.

Hereinafter, examples in which the plurality of objects 410 are converted into a plurality of virtual objects and displayed in a visual image will be described.

FIG. 6 is a diagram 6000 for explaining conversion to an XR environment according to a theme corresponding to the color of the execution screen of an application according to an embodiment of the present disclosure.

(a) of FIG. 6 corresponds to the color of the execution screen of the application displayed on the screen of the electronic device (e.g., the electronic device 102 or 104 of FIG. 1, the electronic device 401 and the electronic device 501 of FIG. 4). This diagram is for explaining a case where a plurality of objects 410 are converted into a plurality of virtual objects 420 when the theme is a bright theme. In (b) of FIG. 6, when the theme corresponding to the color of the execution screen of the application displayed on the screen of the electronic device is a dark theme, a plurality of objects 410 are converted into a plurality of virtual objects 420. This is a drawing to explain.

According to one embodiment, a bright theme may be a case where the brightness of the theme is greater than or equal to a first threshold. A dark theme may be a case where the brightness of the theme is greater than or equal to a second threshold. The first threshold and the second threshold may be the same or different.

Referring to (a) of FIG. 6, a first object 411, a second object 412, and a third object 413 are shown. The first object 411, the second object 412, and the third object 413 may be objects to which a bright theme is applied. The theme of the plurality of virtual objects 420 in the visual image 430 into which the plurality of objects 410 are converted may be determined based on the theme of the plurality of objects 410. For example, since the theme of the plurality of objects 410 is a bright theme, the theme of the plurality of virtual objects 420 may also be determined to be a bright theme.

Referring to (b) of FIG. 6, a first object 411, a second object 412, and a third object 413 are shown. The first object 411, the second object 412, and the third object 413 may be objects to which a dark theme is applied. The theme of the plurality of virtual objects 420 in the visual image 430 into which the plurality of objects 410 are converted may be determined based on the theme of the plurality of objects 410. For example, since the theme of the plurality of objects 410 is a dark theme, the theme of the plurality of virtual objects 420 may also be determined to be a light theme.

According to one embodiment, a processor (e.g., processor 120 of FIG. 1) of a head-worn electronic device (e.g., electronic device 101 of FIG. 1 and head-worn electronic device 201 of FIG. 2) may run an application. You can check information about the theme corresponding to the color of the execution screen. The processor of the head-worn electronic device may determine the theme of the plurality of virtual objects 420 based on information about the theme. The processor of the head-wearable electronic device may determine the theme of the plurality of virtual objects 420 to be the same theme as the theme of the plurality of objects 410 .

According to one embodiment, the processor of the electronic device may check information about the theme corresponding to the color of the execution screen of the application. The processor of the electronic device may determine the theme of the plurality of virtual objects 420 based on information about the theme. The processor of the electronic device may determine the theme of the plurality of virtual objects 420 to be the same theme as the theme of the plurality of objects 410 .

In Figure 6, the description is based on light and dark themes, but it is not limited to this and can also be applied to colored themes.

Hereinafter, a method for controlling the colors of virtual objects by an electronic device and a head-worn electronic device according to the type of visual image 430 will be described.

FIG. 7 is a diagram 700 illustrating a method of converting each screen of an electronic device into a different visual image according to an embodiment of the present disclosure.

Referring to FIG. 7, a first object 411 and a second object are displayed on the screen of an electronic device (e.g., the electronic device 102 or 104 of FIG. 1, the electronic device 401 and the electronic device 501 of FIG. 4). 412 and a third object 413 are shown.

The plurality of objects 410 may be converted into a plurality of virtual objects 420 to be displayed in a visual image (eg, the visual image 430 of FIG. 4). A plurality of converted virtual objects 420 may be placed within the visual image.

At this time, the visual image may be one or a combination of two or more of AR, VST, or VR environments. Accordingly, head-worn electronic devices (e.g., electronic device 102 in FIG. 1 and head-worn electronic device 201 in FIG. 2) may be subject to light interference, unlike electronic devices, in AR and VST environments. Accordingly, when a plurality of virtual objects 420 are displayed with the same brightness and saturation as the objects on the screen of the electronic device, visibility may be lowered.

First, we will describe a case where objects are displayed in the first visual image 701, which is an AR or VST environment in which images captured of the user's surrounding environment are displayed.

The first virtual object 421, the second virtual object 422, and the third virtual object 423 may be objects converted from the plurality of objects 410 to be displayed in the first visual image 701.

According to one embodiment, the processor of the electronic device may identify the arrangement status of the plurality of objects 410. The processor of the electronic device displays a plurality of virtual objects 420 to correspond to the arrangement state of the plurality of objects 410 on the display of the head-worn electronic device (e.g., the display module 160 of FIG. 1 and the display of FIG. 2). It can be placed at (240)).

According to one embodiment, the processor of the head-worn electronic device may identify the arrangement status of the plurality of objects 410. The processor of the head-worn electronic device may arrange a plurality of virtual objects 420 on the display of the head-worn electronic device to correspond to the arrangement status of the plurality of objects 410.

For example, the first virtual object 421, the second virtual object 422, and the third virtual object 423 are arranged in the first visual image 701 to correspond to the arrangement state of the plurality of objects 410. It can be.

According to one embodiment, the processor of the electronic device arranges the plurality of virtual objects 420 in a visual image displayed by the display of the head-worn electronic device in the same arrangement as the arrangement of the objects with the same name on the screen of the electronic device. can do.

According to one embodiment, the processor of the head-worn electronic device displays the plurality of virtual objects 420 in the same arrangement as the arrangement on the screen of the electronic device of the objects having the same name at the time when the display of the head-worn electronic device displays the plurality of virtual objects 420. It can be placed within the video.

For example, the first virtual object 421, the second virtual object 422, and the third virtual object 423 are arranged in the same manner as the arrangement on the screen of the electronic device of the plurality of objects 410 having the same name. It may be placed within the first visual image 701.

At this time, the theme of the plurality of virtual objects 420 may be determined based on the theme corresponding to the color of the application execution screen. When a plurality of virtual objects 420 are arranged in the first visual image 701, visibility may be lowered due to interference of light. Therefore, the colors of objects may need to be adjusted for visibility. Color can be determined by hue, brightness, and saturation.

According to one embodiment, the processor of the head-worn electronic device (eg, processor 120 of FIG. 1) or the processor of the electronic device may control the colors of the plurality of virtual objects 420. The processor of the head-worn electronic device or the processor of the electronic device may control at least one or a combination of brightness and saturation of the plurality of virtual objects 420.

According to one embodiment, the processor of the head-worn electronic device or the processor of the electronic device may increase at least one of the brightness or saturation of the plurality of virtual objects 420 by N% or decrease it by N%. The processor of the head-worn electronic device or the processor of the electronic device may increase the brightness and saturation of the plurality of virtual objects 420 by N% or decrease them by N%, respectively.

According to one embodiment, when it is necessary to display a plurality of virtual objects 420 in the first visual image 701 regardless of the theme of the plurality of virtual objects 420, the processor or electronic device of the head-worn electronic device The processor of the device may increase or decrease at least one of the brightness and saturation of the plurality of virtual objects 420 by N% or a combination thereof by N%, respectively.

According to one embodiment, the processor of the head-worn electronic device or the processor of the electronic device may control the brightness and saturation according to the theme of the plurality of virtual objects 420. When the theme of the virtual objects 420 is a bright theme, the processor of the head-worn electronic device or the processor of the electronic device may increase saturation by N% and decrease brightness by N%. When the theme of the plurality of virtual objects 420 is a dark theme, the processor of the head-worn electronic device or the processor of the electronic device may increase saturation by N% and decrease brightness by N%.

For example, when the visual image is the first visual image 701, the processor of the head-worn electronic device or the processor of the electronic device may increase the brightness of the bright theme objects by 10% and increase the saturation by 10%. You can. When the visual image is the first visual image 701, the processor of the head-worn electronic device or the processor of the electronic device may reduce the brightness of dark-themed objects by 10% and increase the saturation by 10%.

Hereinafter, a case where virtual objects are displayed in the second visual image 702, which is a VR environment that outputs a virtual environment, will be described. Since the VR environment displays a virtual environment, for example, a space made up of computer graphics, it may not be interfered with by light, unlike an AR or VST environment.

Accordingly, the colors of the first virtual object 421, the second virtual object 422, and the third virtual object 423 arranged in the second visual image 702 may not be controlled. The processor of the head-worn electronic device or the processor of the electronic device may display a plurality of virtual objects 420 in the second visual image 702 with the same brightness and saturation as the objects in the screen of the electronic device. That is, the setting values of the electronic device can be displayed as is in the second visual image 702.

Hereinafter, when converting the plurality of objects 410 into a plurality of virtual objects 420, the sizes of fonts and objects included in the plurality of objects 410 will be described.

Figure 8 is a diagram for explaining a method of converting the size of a font and the size of an object according to an embodiment of the present disclosure.

Referring to FIG. 8, the font size of the first object 411 may be 20 dp. The font size of the second object 412 may be 17 dp and 13 dp. The font size of the third object 413 may be 15 dp.

In electronic devices (e.g., electronic device 102 or 104 in FIG. 1, electronic device 401, and electronic device 501 in FIG. 4), the unit of font size may be dp (density independent pixels). A dp is a density-independent pixel, which can be a unit based on the physical density of the screen. However, in the AR environment, VST environment, and VR environment, subjects may be expressed at the same rate as the real world, so units based on physical density may be meaningless. Accordingly, the processor of the head-worn electronic device (e.g., processor 120 in FIG. 1) may convert dp, a unit of fonts included in objects, into mm, a unit of fonts included in virtual objects. According to one embodiment, the processor of the electronic device may convert dp, a unit of fonts included in objects, to mm, a unit of fonts included in virtual objects.

For example, the processor of the electronic device may convert 20 dp, which is the font size of the first object 411, to 8 mm in the first virtual object 421 in the visual image 430. The processor of the electronic device may convert the font sizes of 17 dp and 13 dp of the second object 412 into 8 mm and 7 mm in the second virtual object 422 in the visual image 430. The processor of the electronic device may convert 15 dp, which is the font size of the third object 413, to 8 mm in the third virtual object 423 in the visual image 430.

Additionally, according to one embodiment, the processor of the head-worn electronic device or the processor of the electronic device converts the size of the virtual objects and the sizes included in the virtual objects based on the size of the visual image 430 provided to the user. can do. The visual image 430 may be in any one of VR, AR, and VST environments. Accordingly, the visual image 430 can provide a 3D spatial experience to the user. At this time, when virtual objects are output in VR, AR, and VST environments in the same size as the screen of the electronic device, they may be small and have low visibility. For example, if the user wants to float a plurality of virtual objects 420 in a relatively distant random space within the visual image 430, a plurality of virtual objects (420) are created with the same size as the screen size of the electronic device. When plotting 420, the size of the plurality of virtual objects 420 may be so small that it may be difficult to see.

According to one embodiment, the processor of the head-worn electronic device may enlarge or reduce the size of the objects in proportion to the distance from the user of the area of the visual image 430 in which the plurality of virtual objects 420 are arranged. . The processor of the head-worn electronic device may enlarge or reduce the size of the font included in the virtual object in proportion to the distance from the user of the area of the visual image 430 where the plurality of virtual objects 420 are arranged.

According to one embodiment, the processor of the electronic device may enlarge or reduce the size of the virtual objects 420 in proportion to the distance from the user in the area of the visual image 430 where the plurality of virtual objects 420 are arranged. The processor of the electronic device may enlarge or reduce the size of the font included in the virtual object in proportion to the distance from the user of the area of the visual image 430 in which the plurality of virtual objects 420 are arranged.

According to one embodiment, the color of the font included in the plurality of objects 410 may be applied to the color of the font included in the plurality of virtual objects 420. When a gradient is applied to the font included in the plurality of objects 410, the gradient may also be applied to the font included in the plurality of virtual objects 420. At this time, the rule for applying the gradient applied to the plurality of objects 410 may be applied to the plurality of virtual objects 420 as is.

Below, we will explain the sense of depth according to the depth between virtual objects.

FIG. 9 is a diagram 900 for explaining an object identified according to depth according to an embodiment of the present disclosure.

Referring to FIG. 9, a first object 411 and a second object are displayed on the screen of an electronic device (e.g., the electronic device 102 or 104 of FIG. 1, the electronic device 401 and the electronic device 501 of FIG. 4). 412 and a third object 413 are shown.

The second object 412 includes a background object of the second object 412 (e.g., the background object 503 in FIG. 5), a first detailed object 412-1, and a second detailed object 412-2. can do. The first detailed object 412-1 and the second detailed object 412-2 may include content grouped by different criteria. For example, the first detailed object 412-1 may include content grouped based on today's call transmission and reception history. For example, the second detailed object 412-2 may include content grouped based on yesterday's call transmission and reception history.

The first detailed object 412-1 and the second detailed object 412-2 may be placed on top of the background object of the second object 412. Accordingly, the plurality of objects 410 and the plurality of detailed objects 412-1 and 412-2 are transformed into a plurality of virtual objects 420 and a plurality of detailed virtual objects 422-1 and 422-2. ), the first detailed virtual object 422-1 and the second detailed virtual object 422-2 may be placed on the background virtual object of the second virtual object. According to one embodiment, each virtual object may be identified by depth from other virtual objects. For example, the background virtual object, the first sub-object 422-1, and the second sub-object 422-2 may be distinguished by depth.

For example, a shadow is generated at the bottom of the first detailed virtual object 422-1 and the second detailed virtual object 422-2, so that the first detailed virtual object 422-1 and the second detailed virtual object 422-1 are placed on the background virtual object. A sense of depth can be created as if the virtual object 422-2 is placed. Alternatively, the first detailed virtual object 422-1 and the second detailed virtual object 422-2 may be floated in front of the background virtual object within the visual image 430 and placed at different depths in the direction the user is looking. You can.

In the above, the plurality of virtual objects 420 are processors (e.g., processor 120 of FIG. 1) of head-worn electronic devices (e.g., electronic device 101 and head-worn electronic device 201 of FIG. 1). ) or may be controlled and arranged by a processor of an electronic device (e.g., the electronic device 102 or 104 of FIG. 1, the electronic device 401 and the electronic device 501 of FIG. 4).

Accordingly, the user can experience a three-dimensional effect within the visual image 430 through a sense of depth.

FIG. 10 is a diagram 1000 for explaining an object identified according to a depth by layer according to an embodiment of the present disclosure.

Referring to FIG. 10, a plurality of album folders 1001 and 1002 are displayed on the screen of an electronic device (e.g., the electronic device 102 or 104 of FIG. 1, the electronic device 401 and the electronic device 501 of FIG. 4). 1003) is shown. In one embodiment, the plurality of album folders 1001, 1002, and 1003 are a plurality of objects (e.g., the plurality of objects 410 in FIG. 4) in which content such as photos, images, or videos is stored. This may be an example.

Album folder 1 (1001), album folder 2 (1002), and album folder 3 (1003) may each include content grouped by different criteria. For example, album folder 1 (1001) may be a location where photos taken by the user are automatically saved. Album folder 2 (1002) may be a location where videos captured by the user are automatically saved.

Referring to FIG. 10, album folder 1 (1001) may store 0 content. Album folder 2 (1002) may store 10 contents. Album folder 3 (1003) may store 100 pieces of content.

A plurality of album folders 1001, 1002, and 1003 may be converted into a plurality of virtual album objects 1011, 1012, and 1013 to be displayed in the visual image 430.

According to one embodiment, when the plurality of virtual album objects 1011, 1012, and 1013 include one or more contents, a layer that is displayed overlapping with the virtual album object may be added. For example, the first virtual album object There can be one layer of (1011). The second virtual album object 1012 may have two layers. The third virtual album object 1013 may have three layers. The depth of the plurality of virtual album objects 1011, 1012, and 1013 may be expressed through layers. Album objects 1011, 1012, and 1013 allow the user to intuitively recognize the number of contents stored within the object depending on the number of layers.

According to one embodiment, the processor of the head-worn electronic device (e.g., the electronic device 101 and the head-worn electronic device 201 in FIG. 1) adds a layer when the content included inside the virtual object is two or more. This can create a sense of depth. The processor of the head-worn electronic device can add a layer whenever the content contained inside the virtual object exceeds a threshold value determined for each section.

According to one embodiment, the processor of the electronic device may create a sense of depth by adding a layer when the content included inside the virtual object is two or more. The processor of the electronic device may add a layer whenever the content included inside the virtual object exceeds a threshold determined for each section. Accordingly, virtual objects can be distinguished from other objects through the sense of depth provided by the added layer.

In the above, virtual objects identified through layers according to the number of contents included based on the album folder have been described, but this is only an example and is not limited thereto.

FIG. 11 is a diagram 1100 for explaining content identified according to depth according to an exemplary embodiment of the present disclosure.

Referring to FIG. 11, the first image 1101 and the second image displayed on the screen of the electronic device (e.g., the electronic device 102 or 104 of FIG. 1, the electronic device 401 and the electronic device 501 of FIG. 4) Image 1102, third image 1103, fourth image 1104, and fifth image 1105 are shown. For example, the screen of the electronic device may be an execution screen of a gallery application that can display images.

When converting the screen of the electronic device to display the visual image 430, the second images 1102 to 5th images 1105 may be placed before the first image 1101.

A sense of depth can be created by placing the second to fifth images 1102 to 1105 in front of the first image 1101. Additionally, the second to fifth images 1102 to 1105 may be arranged in an arc shape to create a sense of depth.

According to one embodiment, a processor (e.g., processor 120 of FIG. 1) of a head-worn electronic device (e.g., electronic device 101 and head-worn electronic device 201 of FIG. 1) processes a visual image 430. ), the color of the content can be converted depending on the type. If the visual image 430 is in an AR or VST environment, visibility may be lowered due to light interference. Accordingly, the processor of the head-worn electronic device can control not only the color of virtual objects but also the color of content displayed in the visual image 430. When the visual image 430 is a first visual image (e.g., the first visual image 701 in FIG. 7), the processor of the head-worn electronic device can control the color of the image. The processor of the head-worn electronic device can control the brightness and saturation of the image.

According to one embodiment, the processor of the electronic device may control the color of content according to the type of visual image 430. If the visual image 430 is in an AR or VST environment, visibility may be lowered due to light interference. Accordingly, the processor of the electronic device can control not only the color of the virtual objects but also the color of the content displayed in the visual image 430. When the visual image 430 is a first visual image, the processor of the electronic device can control the color of the image. The processor of the electronic device can control the brightness and saturation of the image.

Below, we will explain when the visual image changes from the second visual image to the first visual image.

FIG. 12 is a diagram 1200 for explaining a virtual object according to a change in a visual image according to an embodiment of the present disclosure.

Referring to FIG. 12 , a plurality of objects 410, a plurality of virtual objects 420 in the second visual image 702, and a plurality of virtual objects 420 in the first visual image 701 are shown. . Referring to FIG. 12 , a plurality of objects 410 may be converted into a plurality of virtual objects 420 in the second visual image 702 . When the visual image changes to the first visual image 701, the colors of the plurality of virtual objects 420 in the second visual image 702 may be controlled.

First, a case where a plurality of objects 410 are converted into a plurality of virtual objects 420 in the second visual image 702 will be described. A plurality of virtual objects 420 may be arranged in the second visual image 702 based on the arrangement status of the plurality of objects 410. At this time, as explained in FIG. 7, the second visual image 702 is a VR environment that outputs a virtual environment, and unlike the first visual image 701, light interference may not occur. Accordingly, the plurality of virtual objects 420 may be determined to have the same color and theme as the application execution screen. The plurality of virtual objects 420 may be determined to have the same color as the plurality of objects 410 . The plurality of virtual objects 420 in the second visual image 702 may be determined to have the same saturation and brightness as the plurality of objects 410 .

This will explain a case where the visual image changes from the second visual image 702 to the first visual image 701 according to the user's command. The visibility of the plurality of virtual objects 420 in the first visual image 701 may be reduced due to light interference. At this time, as described in FIG. 7, the colors of a plurality of objects can be controlled. Accordingly, the colors of the plurality of virtual objects 420 in the first visual image 701 may be controlled compared to the plurality of virtual objects 420 in the second visual image 702. The saturation and brightness of the plurality of virtual objects 420 in the first visual image 701 may be controlled compared to the plurality of virtual objects 420 in the second visual image 702.

Conversely, when the visual image changes from the first visual image 701 to the second visual image 702, the colors of the plurality of virtual objects 420 in the first visual image 701 are changed to the plurality of objects 410. ) can be controlled to be equal to When the visual image changes from the first visual image 701 to the second visual image 702, the saturation and brightness of the plurality of virtual objects 420 in the first visual image 701 are changed to the plurality of objects 410. ) can be controlled to be equal to

In the above, the plurality of virtual objects 420 are processors (e.g., processor 120 of FIG. 1) of head-worn electronic devices (e.g., electronic device 101 and head-worn electronic device 201 of FIG. 1). ) or a processor of an electronic device (e.g., the electronic device 102 or 104 in FIG. 1, the electronic device 401 and the electronic device 501 in FIG. 4).

FIG. 13 is a diagram 1300 for explaining screen expansion in an XR environment according to an embodiment of the present disclosure.

Referring to FIG. 13, a first object included in a plurality of objects 410 of an electronic device (e.g., the electronic device 102 or 104 of FIG. 1, the electronic device 401 and the electronic device 501 of FIG. 4) An object 411, a second object 412, and a third object 413 are shown. The third object 413 may be an extended object that can expand the screen left and right or up and down. Referring to FIG. 13 , a first virtual object 421, a second virtual object 422, and a third virtual object 423 included in the plurality of virtual objects 420 displayed in the visual image 430 are shown. . The third virtual object 423 may be an extended virtual object obtained by converting the third object 413, which is an extended object.

The visual image 430 is a first visual image in an AR or VST environment (e.g., the first visual image 701 in FIG. 7) or a second visual image in a VST environment (e.g., the second visual image 702 in FIG. 7). ) can be. Accordingly, the visual image 430 may display an object within the viewing angle of a user wearing a head-worn electronic device (e.g., the electronic device 101 of FIG. 1 and the head-worn electronic device 201 of FIG. 2). . Accordingly, a head-worn electronic device may display a greater amount or number of content than the screen of the electronic device. Therefore, it may be more efficient to display the extended object (e.g., the third virtual object 423 in FIG. 13) in advance on the visual image 430 rather than displaying it in the visual image 430 through a separate operation.

According to one embodiment, when the menu icon of the first object 411 is selected or the screen of the electronic device is swiped to the right, the third object 413 may be displayed on the screen. The third object 413 may be an extended object. The third object 413 may be displayed on the screen by sliding to the right at the same time as the screen on which the first object 411 and the second object 412 are displayed is pushed to the right. The first object 411, the second object 412, and the third object 413 may be converted into the first virtual object 421, the second virtual object 422, and the third virtual object 423, respectively. . At this time, the third virtual object 423 may be an extended virtual object.

According to one embodiment, a processor (e.g., processor 120 of FIG. 1) of a head-worn electronic device (e.g., electronic device 101 and head-worn electronic device 201 of FIG. 1) or a processor of the electronic device. If the screen is expanded by an extended object when one of the objects is selected by the user, the extended virtual object into which the extended object has been converted can be displayed on the visual image 430 along with the virtual objects into which the object has been converted. . For example, the extended virtual object may be displayed in the visual image 430 together with the first virtual object 421 and the second virtual object 422. Since the extended object slides from left to right when the screen of the electronic device is swiped to the right and is displayed on the left side of the second object 412, the extended virtual object may be placed on the left side of the second virtual object 422.

FIG. 14 is a diagram 1400 for explaining the display of a pop-up object according to an embodiment of the present disclosure.

Referring to FIG. 14, a first object 411, a second object 412, and a third object 413 may be displayed. The third object 413 may be a pop-up object created by a specific event while the first object 411 and the second object 412 are displayed. For example, while the first object 411 and the second object 412 are displayed on the screen, a pop-up object notifying the user of a warning may be created.

The pop-up object may overlap the second object 412 that is being displayed on the screen and be displayed on top of the second object 412 .

Likewise, while the first virtual object 421 and the second virtual object 422, which are converted from the first object 411 and the second object 412, are displayed on the visual image 430, a virtual object pops up due to a specific event. A third virtual object 423 may be displayed. The pop-up virtual object may be a new virtual object created in the visual image 430 by a specific event after the first virtual object 421 and the third virtual object 422 are displayed in the visual image 430.

The pop-up virtual object may be an object converted from a pop-up object to be displayed in the visual image 430. The pop-up virtual object may have the same name as the pop-up object. A pop-up virtual object may be placed based on the placement state of the pop-up object. For example, a pop-up virtual object may be displayed at the bottom, overlapping with a second object.

According to one embodiment, the pop-up virtual object may be a virtual object that displays content grouped based on a different standard from the objects being displayed in the visual image 430. Accordingly, the pop-up virtual object can be distinguished by depth by being displayed floating in front of the first virtual object 421 and the second virtual object 422.

In the above, the plurality of virtual objects 420 are processors (e.g., processor 120 of FIG. 1) of head-worn electronic devices (e.g., electronic device 101 and head-worn electronic device 201 of FIG. 1). ) or a processor of an electronic device (e.g., the electronic device 102 or 104 in FIG. 1, the electronic device 401 and the electronic device 501 in FIG. 4).

FIG. 15 is a flowchart for explaining a method of operating an electronic device according to an embodiment of the present disclosure.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Operations 1501 to 1504 may be performed by an electronic device (e.g., the electronic device 102 or 104 in FIG. 1, the electronic device 401, and the electronic device 501 in FIG. 4). Specifically, operations 1501 to 1504 may be performed by a processor included in the electronic device.

In operation 1501, a plurality of objects related to the execution screen of the application stored in the memory are retrieved from the memory of the electronic device to be displayed on the execution screen of the application being executed by the electronic device (e.g., the plurality of objects in FIG. 4 ( Information containing 410)) can be received.

According to one embodiment, a display (eg, the display 402 in FIG. 4 and the display 502 in FIG. 5) may display an execution screen of an application. The execution screen includes a plurality of objects, and the plurality of objects may display content related to the application.

In operation 1502, a plurality of objects included in the information are displayed on a display (e.g., the display module of FIG. 1) of a head-worn electronic device (e.g., the electronic device 101 and the head-worn electronic device 201 of FIG. 1). To display a visual image (e.g., visual image 430 of FIG. 4) output by 160 and the display 260 of FIG. 2, a plurality of virtual objects (e.g., a plurality of virtual objects of FIG. 4) are displayed. 420)).

In operation 1503, information about the theme corresponding to the color of the application execution screen and information about the visual image output by the display of the head-worn electronic device can be confirmed.

In operation 1504, at least one of the brightness and saturation of a plurality of virtual objects may be controlled based on at least one of information about the theme and information about the visual image.

Since the details described above with reference to FIGS. 1 to 14 are applied to each operation shown in FIG. 15, a more detailed description will be omitted.

According to one embodiment, the execution screen of the application may include a background object of the execution screen (eg, the background object 503 in FIG. 5) and a plurality of objects arranged on the background object. Each object included in the plurality of objects may have a predetermined name, and contents related to the object may be grouped and displayed.

According to one embodiment, an object may include one or more detailed objects (eg, the first detailed object 412-1 in FIG. 9 and the second detailed object 412-2 in FIG. 9) related to the object. The virtual object may include one or more detailed virtual objects (eg, the first detailed virtual object 422-1 in FIG. 9 and the second detailed virtual object 422-2 in FIG. 9) related to the virtual object.

According to one embodiment, the processor may identify the arrangement status of a plurality of objects from the execution screen of an application running on the electronic device. The processor may arrange a plurality of virtual objects in a visual image displayed by the display of the head-worn electronic device to correspond to the placement state.

According to one embodiment, the processor may arrange the virtual object in a visual image displayed by the display of the head-worn electronic device in the same arrangement as the arrangement of the object with the same name on the screen of the electronic device.

According to one embodiment, the processor may convert the sizes of the plurality of virtual objects and the sizes of fonts included in the plurality of virtual objects based on the size of the visual image provided to the user.

According to one embodiment, the processor may determine a theme corresponding to the color of a plurality of virtual objects displayed on the display of the head-worn electronic device to match the theme corresponding to the color of the execution screen of the application.

According to one embodiment, the visual image is a first visual image that outputs the user's surrounding environment captured by the camera of the head-worn electronic device (e.g., the first visual image 701 in FIG. 7) or a virtual environment that outputs. It may include a second visual image (e.g., the second visual image 702 in FIG. 7). The processor may determine whether the display of the head-worn electronic device outputs a first visual image or a second visual image. The processor may control at least one of the brightness and saturation of the plurality of virtual objects according to the output visual image.

According to one embodiment, each virtual object included in the plurality of virtual objects is displayed on a display of a head-worn electronic device such that content related to the virtual object is grouped and displayed and identified from other virtual objects by depth. It can be displayed in .

According to one embodiment, when any one virtual object among the plurality of virtual objects includes a plurality of contents, any one virtual object among the plurality of virtual objects is a virtual object among the plurality of virtual objects. A layer that overlaps the object can be added. One virtual object among a plurality of virtual objects may be distinguished from other virtual objects by a layer.

According to one embodiment, when the display of the head-worn electronic device converts the screen from the second visual image to the first visual image or converts the screen from the first visual image to the second visual image, the processor displays a plurality of screens. At least one of the brightness and saturation of virtual objects can be controlled.

According to one embodiment, when displaying a new virtual object within the visual image of the display of the head-worn electronic device after a plurality of virtual objects have been displayed within the visual image of the display of the head-worn electronic device, the processor may display the new virtual object within the visual image of the display of the head-worn electronic device. can be identified using a plurality of virtual objects displayed on the display and depth.

FIG. 16 is a flowchart for explaining a method of operating a head-worn electronic device according to an embodiment of the present disclosure.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Operations 1601 to 1604 may be performed by a head-worn electronic device (eg, the electronic device 101 and the head-worn electronic device 201 of FIG. 1). Specifically, operations 1601 to 1604 may be performed by a processor included in the head-worn electronic device (eg, processor 120 of FIG. 1).

In operation 1601, the processor of the head-worn electronic device processes an application executing on the electronic device (e.g., electronic device 102 or 104 in FIG. 1, electronic device 401, and electronic device 501 in FIG. 4). Information including a plurality of objects (eg, a plurality of objects 410 in FIG. 4) related to the execution screen may be received.

According to one embodiment, the display of the electronic device (eg, the display 402 of FIG. 4 and the display 502 of FIG. 5) may display an application execution screen. The execution screen may include multiple objects. A plurality of objects may display content related to an application.

In operation 1602, the processor of the head-worn electronic device displays a plurality of objects included in the information on the display of the head-worn electronic device (e.g., the display module 160 of FIG. 1 and the display 260 of FIG. 2). The visual image output (e.g., visual image 430 of FIG. 4) may be converted into a plurality of virtual objects (e.g., plural virtual objects 420 of FIG. 4) for display.

In operation 1603, the processor of the head-worn electronic device may check information about the theme corresponding to the color of the execution screen of the application and information about the visual image output by the display of the head-worn electronic device.

In operation 1604, the processor of the head-worn electronic device may control at least one of the brightness and saturation of the plurality of virtual objects based on at least one of information about the theme and information about the visual image.

Since the details described above with reference to FIGS. 1 to 14 are applied to each operation shown in FIG. 16, detailed descriptions are omitted.

According to one embodiment, the execution screen of the application may include a background object of the execution screen (eg, the background object 503 in FIG. 5) and a plurality of objects arranged on the background object. Each object included in the plurality of objects may have a predetermined name, and contents related to the object may be grouped and displayed.

According to one embodiment, an object may include one or more detailed objects (eg, the first detailed object 412-1 in FIG. 9 and the second detailed object 412-2 in FIG. 9) related to the object. The virtual object may include one or more detailed virtual objects (eg, the first detailed virtual object 422-1 in FIG. 9 and the second detailed virtual object 422-2 in FIG. 9) related to the virtual object.

According to one embodiment, the processor of the head-worn electronic device may identify the arrangement status of a plurality of objects from the execution screen of an application that the electronic device is executing. The processor of the head-worn electronic device may arrange a plurality of virtual objects in a visual image displayed by the display of the head-worn electronic device to correspond to the arrangement state.

According to one embodiment, the processor of the head-worn electronic device may determine a theme corresponding to the color of a plurality of virtual objects displayed on the display of the head-worn electronic device to match the theme corresponding to the color of the execution screen of the application. there is.

According to one embodiment, the processor of the head-worn electronic device may arrange virtual objects in a visual image displayed by the display of the head-worn electronic device in the same arrangement as the arrangement of objects with the same name on the screen of the electronic device. there is.

According to one embodiment, the processor of the head-worn electronic device may convert the size of the plurality of virtual objects and the size of the font included in the plurality of virtual objects based on the size of the visual image provided to the user. there is.

According to one embodiment, the processor of the head-worn electronic device determines a theme corresponding to the color of the plurality of virtual objects displayed on the display of the head-worn electronic device to match the theme corresponding to the color of the execution screen of the application. You can.

According to one embodiment, the processor of the head-worn electronic device may display the first visual image (e.g., the first visual image 701 of FIG. 7) or the second visual image (e.g., FIG. 7) of the head-worn electronic device. It can be determined whether the second visual image 702 of 7 is output. The processor of the head-worn electronic device may control at least one of the brightness and saturation of a plurality of virtual objects according to the output visual image.

According to one embodiment, each virtual object included in the plurality of virtual objects is displayed on a display of a head-worn electronic device such that content related to the virtual object is grouped and displayed and identified from other virtual objects by depth. It can be displayed in .

According to one embodiment, when any one virtual object among the plurality of virtual objects includes a plurality of contents, any one virtual object among the plurality of virtual objects is a virtual object among the plurality of virtual objects. A layer that overlaps the object can be added. One virtual object among a plurality of virtual objects may be distinguished from other virtual objects by a layer.

According to one embodiment, the processor of the head-worn electronic device causes the display of the head-worn electronic device to convert the screen from the second visual image to the first visual image, or convert the screen from the first visual image to the second visual image. When converting, at least one of the brightness and saturation of a plurality of virtual objects can be controlled.

According to one embodiment, the processor of the head worn electronic device generates a new virtual object within the visual image of the display of the head worn electronic device after a plurality of virtual objects have been displayed in the visual image of the display of the head worn electronic device. When displaying, a new object can be identified using the depth and a plurality of virtual objects being displayed on the display.

In addition, the embodiments of the present invention disclosed in the specification and drawings are merely provided as specific examples to easily explain the technical content according to the embodiments of the present invention and to aid understanding of the embodiments of the present invention, and limit the scope of the embodiments of the present invention. It is not intended to be limiting. Therefore, the scope of the various embodiments of the present invention should be interpreted as including all changes or modified forms derived based on the technical idea of the various embodiments of the present invention in addition to the embodiments disclosed herein. .

Claims (20)

In the electronic device (102, 104; 401; 501),
The electronic devices (102, 104; 401; 501) are connected to a head-worn electronic device (101; 201),
Memory;
display(402; 502); and
Information including a plurality of objects 410 related to the execution screen of the application stored in the memory to be displayed on the execution screen of the application being executed by the electronic device (102, 104; 401; 501) from the memory. - The display (402; 502) displays an execution screen of the application, the execution screen includes the plurality of objects 410, and the plurality of objects 410 are related to the application. Displays contents -,
A plurality of virtual objects 420 are used to display the plurality of objects 410 included in the information on the visual image 430 output by the display 160; 240 of the head-worn electronic device 101; 201. ) and convert to
Check information about the theme corresponding to the color of the execution screen of the application and information about the visual image 430 output by the display 160; 240 of the head-worn electronic device 101; 201,
A processor that controls at least one of brightness and saturation of the plurality of virtual objects 420 based on at least one of information about the theme and information about the visual image.
Electronic devices containing.
According to paragraph 1,
The execution screen of the application is,
Includes a background object 503 of the execution screen and a plurality of objects 410 arranged on the background object 503,
Each of the objects 411, 412, and 413 included in the plurality of objects 410,
An electronic device whose name is predetermined and which groups and displays contents related to the objects 411, 412, and 413.
According to any one of claims 1 and 2,
Objects 411, 412, and 413 included in the plurality of objects 410 are:
Contains one or more detailed objects (412-1, 412-2) related to the objects (411, 412, 413),
Virtual objects 421, 422, and 423 included in the plurality of virtual objects 420 are:
An electronic device including one or more detailed virtual objects (422-1, 422-2) related to the virtual objects (421, 422, 423).
According to any one of claims 1 to 3,
The processor,
The electronic device (102, 104; 401; 501) identifies the placement status of the plurality of objects 410 from the execution screen of the application being executed, and creates the plurality of virtual objects 420 to correspond to the placement status. An electronic device disposed within a visual image (430) displayed by a display (160; 240) of the head-worn electronic device (101; 201).
According to any one of claims 1 to 4,
The processor,
Virtual objects 421, 422, and 423 are displayed on the head-worn electronic device ( An electronic device disposed within the visual image 430 displayed by the display 160; 240 of 101; 201.
According to any one of claims 1 to 5,
The processor,
An electronic device that converts the size of the plurality of virtual objects 420 and the size of a font included in the plurality of virtual objects 420 based on the size of the visual image 430 provided to the user.
According to any one of claims 1 to 6,
The processor,
A theme corresponding to the color of the plurality of virtual objects 420 displayed on the display 160; 240 of the head-worn electronic device 101; 201 to match the theme corresponding to the color of the execution screen of the application. Electronic device for making decisions.
According to any one of claims 1 to 7,
The visual image 430 is,
It includes a first visual image 701 that outputs the user's surrounding environment captured by the camera (180; 260) of the head-worn electronic device, or a second visual image 702 that outputs a virtual environment,
The processor,
A visual image 430 determines whether the display 160; 240 of the head-worn electronic device 101; 201 outputs the first visual image 701 or the second visual image 702, and outputs the visual image 430. ), controlling at least one of the brightness and saturation of the plurality of virtual objects 420 according to the electronic device.
According to any one of claims 1 to 8,
Each virtual object 421, 422, and 423 included in the plurality of virtual objects 420,
The display (160; 240) of the head-worn electronic device (101; 201) groups and displays content related to the virtual objects (421, 422, 423) and distinguishes them from other virtual objects by depth. displayed in electronic devices.
According to any one of claims 1 to 9,
If one of the plurality of virtual objects 420 includes a plurality of contents,
One virtual object among the plurality of virtual objects 420 is,
An electronic device in which a layer overlapping with any one of the plurality of virtual objects 420 is added and is distinguished from other virtual objects by the layer.
According to any one of claims 1 to 10,
The processor,
The display (160; 240) of the head-worn electronic device (101; 201) converts the screen from the second visual image (702) to the first visual image (701), or changes the screen to the first visual image (701). When converting the screen from the second visual image 702 to the second visual image 702, the electronic device controls at least one of brightness and saturation of the plurality of virtual objects 420.
According to any one of claims 1 to 11,
The processor,
After the plurality of virtual objects 420 are displayed in the visual image 430 of the display 160; 240 of the head worn electronic device 101; 201, a new virtual object is displayed on the head worn electronic device (101; 201). When displaying the new object in the visual image 430 of the display 160; 240 of 101; 201, the plurality of virtual objects 420 and depth that are being displayed on the display 160; 240 An electronic device that identifies using.
In the head-worn electronic device (101; 201),
camera(180; 260);
display(160; 240); and
Controls the camera 180; 260 to capture the user's surrounding environment, and outputs a first visual image 701 or a virtual environment that outputs the user's surrounding environment captured by the camera 180; 260. A processor (120) that controls the display (160; 240) to display a visual image (702)
Including,
The processor 120,
Receiving information including a plurality of objects 410 related to an execution screen of a running application from an electronic device (102, 104; 401; 501), and - displaying the display of the electronic device (102, 104; 401; 501) (402; 502) displays an execution screen of the application, the execution screen includes the plurality of objects 410, and the plurality of objects 410 display content related to the application -,
A plurality of virtual objects ( 420),
Check information about the theme corresponding to the color of the execution screen of the application and information about the visual image 430 output by the display 160; 240 of the head-worn electronic device 101; 201,
A head-worn electronic device that controls at least one of brightness and saturation of the plurality of virtual objects 420 based on at least one of information about the theme and information about the visual image.
According to clause 13,
The execution screen of the application is,
Includes a background object 503 of the execution screen and a plurality of objects 410 arranged on the background object 503,
Each of the objects 411, 412, and 413 included in the plurality of objects 410,
A head-worn electronic device whose name is predetermined and which groups and displays content related to the object (411, 412, 413).
According to any one of claims 13 to 14,
Objects 411, 412, and 413 included in the plurality of objects 410 are:
Contains one or more detailed objects (412-1, 412-2) related to the objects (411, 412, 413),
Virtual objects 421, 422, and 423 included in the plurality of virtual objects 420 are:
A head-worn electronic device comprising one or more detailed virtual objects (422-1, 422-2) related to the virtual objects (421, 422, 423).
According to any one of claims 13 to 15,
The processor 120,
The electronic device (102, 104; 401; 501) identifies the placement status of the plurality of objects 410 from the execution screen of the application being executed, and creates the plurality of virtual objects 420 to correspond to the placement status. A head-worn electronic device disposed within a visual image (430) displayed by a display (160; 240) of the head-worn electronic device (101; 201).
According to any one of claims 13 to 16,
The processor 120,
A theme corresponding to the color of the plurality of virtual objects 420 displayed on the display 160; 240 of the head-worn electronic device 101; 201 to match the theme corresponding to the color of the execution screen of the application. A head-worn electronic device that makes decisions.
According to any one of claims 13 to 17,
The processor 120,
A visual image 430 determines whether the display 160; 240 of the head-worn electronic device 101; 201 outputs the first visual image 701 or the second visual image 702, and outputs the visual image 430. ) A head-worn electronic device that controls at least one of the brightness and saturation of the plurality of virtual objects 420 according to ).
According to any one of claims 13 to 18,
The processor 120,
The display (160; 240) of the head-worn electronic device (101; 201) converts the screen from the second visual image (702) to the first visual image (701), or changes the screen to the first visual image (701). A head-worn electronic device that controls at least one of brightness and saturation of the plurality of virtual objects 420 when converting the screen from the second visual image 702 to the second visual image 702.
In a method of operating an electronic device (102, 104; 401; 501),
An execution screen of the application stored in the memory to be displayed on the execution screen of the application being executed by the electronic device (102, 104; 401; 501) from the memory of the electronic device (102, 104; 401; 501) and An operation of receiving information including a plurality of related objects 410 - the display 402; 502 of the electronic device 102, 104; 401; 501 displays an execution screen of the application, and the execution screen includes comprising the plurality of objects 410, wherein the plurality of objects 410 display content related to the application;
A plurality of virtual objects 420 are used to display the plurality of objects 410 included in the information on the visual image 430 output by the display 160; 240 of the head-worn electronic device 101; 201. ), the action of converting to );
Checking information about a theme corresponding to the color of the execution screen of the application and information about the visual image 430 output by the display 160; 240 of the head-worn electronic device 101; 201; and
An operation of controlling at least one of brightness and saturation of the plurality of virtual objects 420 based on at least one of information about the theme and information about the visual image.
An operation method comprising:

Images