KR20230040232A - Method and device to display extended screen of mobile device - Google Patents

Abstract

According to one embodiment of the present invention, an electronic device is configured to: establish wireless communication with an external device; determine, on the basis of a wireless communication signal of the external device, a display reference position at which to display an extended screen of a mobile device; and provide a user first with the portion of the screen in the field of view, when only a portion of the extended screen is in a field of view. The electronic device includes: a wireless communication module for wireless communication with an external device; an augmented reality (AR) display configured to output an extended screen of a mobile device; a memory configured in which computer-executable instructions are stored; and a processor configured to access the memory and execute the instructions.

Description

Apparatus and method for displaying an extended screen of a mobile device {METHOD AND DEVICE TO DISPLAY EXTENDED SCREEN OF MOBILE DEVICE}

The disclosure below relates to providing augmented reality for an extended screen of a mobile device.

Wearable electronic devices providing augmented reality (AR) services are being released. An augmented reality service is a service that superimposes a virtual image having additional information on a real world image viewed by a user, and can provide a user with a virtual object image including content related to a real object identified from the real world image. there is. A wearable electronic device providing an augmented reality service may have, for example, a head mounted display (HMD).

An electronic device according to an embodiment may provide an extended screen of a mobile device based on wireless communication instead of a vision sensor.

An electronic device according to an embodiment may provide an extended screen of a mobile device without recognizing a plane and a marker in space.

An electronic device according to an embodiment may provide an extended screen of a mobile device in an augmented reality environment without environment and device limitations using wireless communication.

An electronic device according to an embodiment includes a wireless communication module for wireless communication with an external device; An augmented reality display module that outputs an extended screen of a mobile device; a memory in which computer-executable instructions are stored; and a processor that accesses the memory and executes the commands, wherein the commands perform wireless communication with the external device for displaying a reference position in an augmented reality environment to provide the extended screen received from the mobile device. In response to a case where at least some screens of the extended screen to be provided at the display reference location are included in a field of view (FOV) area of the augmented reality display module, the at least some screens are displayed. It can be set to provide to the user through the augmented reality display module.

A computer-implemented method according to an embodiment includes an operation of determining a display reference position to provide an extended screen of a mobile device in an augmented reality environment based on wireless communication with an external device; and in response to a case where at least some screens of the extended screen to be provided at the display reference position are included within a field of view (FOV) area of the augmented reality display module, the at least some screens are displayed through the augmented reality display module. It may include an action provided to the user.

An electronic device according to an embodiment may save real-time computing resources by determining a display reference location based on an external device based on wireless communication.

An electronic device according to an embodiment does not require recognition of a visual marker through wireless communication, and can output within an augmented reality environment even if only a part of the extended screen is included within the viewing angle.

An electronic device according to an embodiment detects a relative positional change between an external device and an electronic device with an average latency time of 50 μs corresponding to a frequency higher than 60 Hz to 120 Hz through ultrawide band (UWB) communication. By tracking, it is possible to provide extended screen content without interruption naturally in the electronic device.

1 is a block diagram illustrating a configuration of an electronic device in a network environment according to an exemplary embodiment.
2 is a diagram illustrating a structure of a wearable electronic device according to an exemplary embodiment.
3 illustrates an exemplary software architecture of an augmented reality system according to one embodiment.
4 is a flowchart illustrating a method of providing an extended screen according to an exemplary embodiment.
5 to 9 are diagrams illustrating examples of providing an extended screen based on a mobile device according to an exemplary embodiment.
10 illustrates an example of providing an extended screen when a mobile device is hidden according to an embodiment.
11 to 15 are diagrams illustrating examples of providing an extended screen based on a wireless communication device according to an embodiment.
16 is a diagram for explaining provision of a part of an extended screen according to a viewing angle of an electronic device according to an exemplary embodiment.
17 is a diagram illustrating rotation tracking using an inertial sensor according to an exemplary embodiment.
18 to 21 are views illustrating control of an extended screen based on manipulation of a mobile device according to an exemplary embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, a sound output module 155, a display 160, an audio module 170, and a sensor module 176. ), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, Alternatively, the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) may be integrated into a single component (eg, display 160). can

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor. For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display 160, the sensor module 176, or the communication module 190) It may control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 180 or communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to one embodiment, the display 160 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or fingerprint sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, a legacy communication module). It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 is a peak data rate for eMBB realization (eg, 20 Gbps or more), a loss coverage for mMTC realization (eg, 164 dB or less), or a U-plane latency for URLLC realization (eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported. The wireless communication module 192 may perform wireless communication in an ultra-wideband (UWB).

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . The external electronic device 102 may be, for example, the wearable electronic device 200 .

According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology. In this specification, an example in which the external electronic device 104 is a wireless communication device mainly having a display module is described, but is not limited thereto.

2 is a diagram illustrating a structure of a wearable electronic device according to an embodiment.

Referring to FIG. 2 , a wearable electronic device 200 (eg, the electronic device 101 or the electronic device 102 of FIG. 1 ) is worn on a user's face and provides an augmented reality service and/or a virtual reality service to the user. Related images can be provided.

In an embodiment, the wearable electronic device 200 includes a first display 205, a second display 210, screen display units 215a and 215b, input optical members 220a and 220b, and a first transparent member 225a. ), a second transparent member 225b, lighting units 230a and 230b, a first PCB 235a, a second PCB 235b, a first hinge 240a, a second hinge 240b, and a photographing camera 245, a plurality of microphones (eg, a first microphone 250a, a second microphone 250b, a third microphone 250c), a plurality of speakers (eg, a first speaker 255a, a second speaker 255b) )), a battery 260, a first recognition camera 265a, a second recognition camera 265b, a first eye detection camera 270a, and a second eye detection camera 270b. can do.

In one embodiment, the augmented reality display module (eg, the first display 205 and the second display 210) is, for example, a liquid crystal display (LCD), a digital mirror display device (digital mirror display) device, DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not shown in the drawings, when the displays 205 and 210 are made of one of a liquid crystal display, a digital mirror display, and a silicon liquid crystal display, the wearable electronic device 200 radiates light to the screen output area of the displays 205 and 210. It may include a light source that In another embodiment, when the display 205 or 210 can generate light by itself, for example, if it is made of either an organic light emitting diode or a micro LED, the wearable electronic device 200 does not include a separate light source, but the user It is possible to provide a virtual image of good quality to the user. In one embodiment, if the displays 205 and 210 are implemented with organic light emitting diodes or micro LEDs, since a light source is unnecessary, the wearable electronic device 200 can be lightened in weight. Hereinafter, the displays 205 and 210 capable of generating light by themselves are referred to as self-luminous displays, and will be described on the premise of self-luminous displays. For reference, as will be described later, the augmented reality display module may output an extended screen of the mobile device.

The displays 205 and 210 according to various embodiments of the present invention may include at least one micro light emitting diode (LED). For example, micro LEDs can express red (R, red), green (G, green), and blue (B, blue) by self-emission, and are small (eg, 100 μm or less), so one chip can A pixel (e.g., one of R, G, and B) can be implemented. Accordingly, when the displays 205 and 210 are composed of micro LEDs, high resolution can be provided without a back light unit (BLU). However, it is not limited thereto, and one pixel may include R, G, and B, and one chip may be implemented with a plurality of pixels including R, G, and B. The displays 205 and 210 may also be called 'light source units'.

In one embodiment, the displays 205 and 210 may include pixels for displaying virtual images. Displays 205 and 210 may further include infrared pixels that emit infrared light.

Depending on the embodiment, the displays 205 and 210 include light-receiving pixels (eg, photo sensor pixels) disposed between the pixels that receive light reflected from the user's eyes, convert it into electrical energy, and output the light. can include more. The light-receiving pixel(s) may be referred to as a 'gaze tracking sensor'. The gaze tracking sensor may detect infrared light emitted by infrared pixels included in the displays 205 and 210 and reflected by the user's eyes.

The wearable electronic device 200 may detect the user's gaze direction (eg, pupil movement) through light-receiving pixels. For example, the wearable electronic device 200 provides a visual direction for the user's right eye and a user's left eye through one or more light-receiving pixels constituting the first display 205 and one or more light-receiving pixels constituting the second display 210. It is possible to detect and track the gaze direction for . The wearable electronic device 200 may determine the location of the center of the virtual image according to directions of gazes of the user's right and left eyes detected through one or more light-receiving pixels (eg, directions in which the pupils of the user's right and left eyes gaze). .

The wearable electronic device 200 may include displays 205 and 210, a first transparent member 225a and/or a second transparent member 225b, and a user wears the wearable electronic device 200 on his or her face. can be used According to one embodiment, the first transparent member 225a may be disposed to face the user's right eye, and the second transparent member 225b may be disposed to face the user's left eye. According to various embodiments, when the display 205 or 210 is transparent, it may be disposed at a position facing the user's eyes to form the screen display units 215a and 215b.

The first display 205 and the second display 210 may each include a first control circuit (not shown). The first control circuit may control the first display 205 and the second display 210 . The first control circuit may control the operation of the liquid crystal element of the transparent cover (not shown) included in the first display 205 and the second display 210 . In one embodiment, the light emitted from the displays 205 and 210 passes through a lens (not shown) and a waveguide, and is formed on the first transparent member 225a disposed to face the user's right eye. 215a and the screen display units 215b formed on the second transparent member 225b disposed to face the user's left eye.

Lenses (not shown) may be disposed in front of the displays 205 and 210 . The lens (not shown) may include a concave lens and/or a convex lens. For example, the lens (not shown) may include a projection lens or a collimation lens (not shown).

In an embodiment, light emitted from the displays 205 and 210 may be guided to a light path to a waveguide through the input optical members 220a and 220b. Light moving inside the waveguide may be guided toward the user's eyes through the output optical member. The screen display units 215a and 215b may be determined based on light emitted in the direction of the user's eyes.

For example, light emitted from the displays 205 and 210 may be reflected on the grating area of the waveguide formed on the input optical members 220a and 220b and the screen display units 215a and 215b and then be transmitted to the user's eyes. .

In one embodiment, the screen display units 215a and 215b or transparent members (eg, the first transparent member 225a and the second transparent member 225b) may include a lens including a wave guide or a reflective lens. there is. The waveguide serves to deliver the light source generated by the displays 205 and 210 to the user's eyes, and may be called an 'optical waveguide' or 'optical waveguide'. Hereinafter, 'optical waveguide' or 'waveguide' may correspond to the screen display units 215a and 215b.

The screen display units 215a and 215b are paths through which external light is incident, totally reflected, and emitted, and can be distinguished from the first transparent member 225a and the second transparent member 225b through which external light is simply reflected or transmitted. there is.

The screen display units 215a and 215b may be made of glass, plastic, or polymer, and may include a nanopattern formed on an inner or outer surface, for example, a polygonal or curved grating structure, that is, a lattice. structure may be included. According to an embodiment, light incident to one end of the screen display units 215a or 215b through the input optical members 220a or 220b may be propagated inside the display optical waveguide by nano-patterns and provided to the user. In one embodiment, the screen display units 215a and 215b configured as free-form prisms may provide incident light to a user through a reflection mirror.

The screen display units 215a and 215b may include at least one of a diffractive element (eg, a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (eg, a reflective mirror). The screen display units 215a and 215b use at least one diffractive element or reflective element included in the screen display units 215a and 215b to obtain information from the display (eg, the first display 205 and the second display 210). The emitted display light may be directed to the user's eyes.

According to various embodiments, the diffractive element may include input optical members 220a and 220b/output optical members 220a and 220b. For example, the input optical members 220a and 220b may mean an input grating area, and the output optical member may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from the displays 205 and 210 (eg, micro LEDs) to transfer the light to the screen display units 215a and 215b. The output grating area may serve as an outlet for diffracting (or reflecting) light transmitted to the waveguide to the user's eyes.

According to various embodiments, the reflective element may include a total internal reflection optical element or total internal reflection waveguide for total internal reflection (TIR). For example, total reflection is a method of inducing light, and light (eg, virtual image) input through the input grating area is 100% or 100% on a part (eg, a specific surface) of the screen display units 215a and 215b. It can mean making the incident angle so that it is reflected close to , so that it is transmitted 100% or close to 100% to the output grating area.

The first transparent member 225a and/or the second transparent member 225b may be formed of a glass plate, a plastic plate, or a polymer, and may be transparent or translucent. According to the embodiment, the first transparent member 225a may be disposed to face the user's right eye, and the second transparent member 225b may be disposed to face the user's left eye.

The lighting units 230a and 230b may have various uses depending on the position where they are attached. For example, the lighting units 230a and 230b may be attached around a frame of the wearable electronic device 200 . The lighting units 230a and 230b may be used as auxiliary means for facilitating eye gaze detection when pupils are photographed by the eye tracking cameras 270a and 270b. The lighting units 230a and 230b may use an IR LED (infra-red light emitting device) having a visible light wavelength or an infrared wavelength.

Alternatively, the lighting units 230a and 230b may be hinged (eg, a first hinge) connecting a frame (eg, a rim) of the wearable electronic device 200 and a temple corresponding to the temple of the glasses. (240a), the second hinge (240b), or the surroundings of the camera (eg, the first recognition camera 265a, the second recognition camera 265b) mounted adjacent to the bridge connecting the frame can be attached to In this case, the cameras 265a and 265b may be, for example, global shutter (GS) cameras, but are not necessarily limited thereto.

In this case, the lighting units 230a and 230b may be used as means for supplementing ambient brightness when shooting with a global shutter (GS) camera. The lighting units 230a and 230b may be used when it is not easy to detect a subject to be photographed in a dark environment or due to mixed and reflected light of various light sources.

Depending on the embodiment, the lighting units 230a and 230b may be omitted. The lighting units 230a and 230b may be replaced by infrared pixels included in the first display 205 and the second display 210 . In another embodiment, the lighting units 230a and 230b may be included in the wearable electronic device 200 to assist infrared pixels included in the first display 205 and the second display 210 .

A printed circuit board (PCB) (eg, the first PCB 235a and the second PCB 235b) may be disposed on the temple portion of the wearable electronic device 200, and each module is provided through a flexible printed circuit board (FPCB). (e.g. camera, display, audio, or sensor) and other printed circuit boards. According to various embodiments, the at least one printed circuit board may include a first printed circuit board 235a, a second board 235b, and an interposer disposed between the first board 235a and the second board 235b. (not shown).

In one embodiment, components constituting the wearable electronic device 200 except for the first display 205 and the second display 210 are included in the PCB (eg, the first PCB 235a and the second PCB 235b). A control circuit (not shown) for controlling may be located. The control circuit may control components other than the first display 205 and the second display 210 and perform an operation such as estimating a depth value. The control circuit may include a wireless communication module (eg, the communication module 190 of FIG. 1 ) or a memory (eg, the memory 130 of FIG. 1 ). The control circuit may control the first display 205, the second display and/or other components. The wireless communication module may illustratively establish wireless communication with other external devices (eg, the electronic device 101 or the electronic device 104 of FIG. 1 ) in the ultra-wideband. However, the communication band of the wireless communication module is not limited thereto.

The hinges 240a and 240b may correspond to portions where a temple of the wearable electronic device 200 and a frame (eg, a rim) are coupled.

In one embodiment, the photographing camera 245 may be referred to as a high resolution (HR) or photo video (PV) camera, and may include a high resolution camera. The photographing camera 245 may include a color camera equipped with functions for obtaining high-quality images, such as an auto focus (AF) function and an optical image stabilizer (ois). The photographing camera 245 is not limited thereto, and may include a global shutter (GS) camera or a rolling shutter (RS) camera.

In one embodiment, a plurality of microphones (eg, the first microphone 250a, the second microphone 250b, and the third microphone 250c) may process external sound signals into electrical voice data. The processed voice data may be utilized in various ways according to a function (or an application being executed) being performed in the wearable electronic device 200 .

In one embodiment, a plurality of speakers (eg, first speaker 255a, second speaker 255b) is received from a communication circuit (eg, communication module 190 of FIG. 1) or memory (eg, FIG. 1 Audio data stored in the memory 130 may be output.

In one embodiment, one or more batteries 260 may be included and may supply power to components constituting the wearable electronic device 200 .

In an embodiment, the first recognition camera 265a and the second recognition camera 265b may perform 3 degrees of freedom (3DoF), 6DoF head tracking, and hand detection and tracking. , a camera used for gesture and/or spatial recognition. For example, the first recognition camera 265a and the second recognition camera 265b may include a global shutter (GS) camera to detect and track the movement of the head and hand. For example, since a stereo camera can be used for head tracking and spatial recognition, two global shutter (GS) cameras of the same specifications and performance can be used, detecting fast hand motions and fine movements such as fingers, and detecting motions. A rolling shutter (RS) camera may be used for tracking. In one embodiment, a global shutter (GS) camera with excellent camera performance (eg, image dragging) may be mainly used, but is not necessarily limited thereto. According to various embodiments, a rolling shutter (RS) camera may be used. The first recognition camera 265a and the second recognition camera 265b may perform a SLAM (simultaneous localization and mapping) function through spatial recognition and depth imaging for 6DoF. Also, the first recognition camera 265a and the second recognition camera 265b may perform a user gesture recognition function.

In one embodiment, at least one sensor (not shown) (eg, a gyro sensor, an acceleration sensor, a geomagnetic sensor, and/or a gesture sensor), a first recognition camera 265a, and a second recognition camera 265b Performs at least one of head tracking for 6DoF, pose estimation & prediction, gesture and / or spatial recognition, and / or slam function through depth shooting can

In another embodiment, the first recognition camera 265a and the second recognition camera 265b may be used separately as a camera for head tracking and a camera for hand tracking.

In an embodiment, the first eye tracking camera 270a and the second eye tracking camera 270b may detect and track a pupil. In the first eye tracking camera 270a and the second eye tracking camera 270b, the center of the virtual image projected on the wearable electronic device 200 is in the direction in which the pupil of the user wearing the wearable electronic device 200 gazes. It can be used for positioning. Global shutter (GS) cameras may be mainly used as the first eye tracking camera 270a and the second eye tracking camera 270b, for example, to detect pupils and track fast pupil movements. The first eye tracking camera 270a may correspond to the right eye of the user, and the second eye tracking camera 270b may correspond to the left eye of the user. In this case, camera performance and specifications of the first eye tracking camera 270a and the second eye tracking camera 270b may be the same, but are not necessarily limited thereto.

3 illustrates an exemplary software architecture of an augmented reality system according to one embodiment.

Hereinafter, in this specification, the electronic device 310 may mainly refer to the wearable electronic device 200 (eg, the electronic device 102 of FIG. 1 ). The mobile device 320 is a device providing an extended screen and may mainly represent the electronic device 101 of FIG. 1 . The external device 330 may be a device that establishes communication with the electronic device 310 and provides information about a reference display location, which is a location to provide an extended screen to the electronic device 310 .

The software architecture of the mobile device 320 may include an extended screen application 321, a virtual display manager 322, an operating system 323, an application manager 324, a network interface 325, and a video transmission service 326. can

The extended screen application 321 may display applications, icons, and notifications executed on the mobile device 320 on a plane corresponding to the extended screen. The extended screen may also be exemplarily referred to as a DeX (Desktop eXperience) screen. The extended screen is a screen output from other display modules other than the display module itself of the mobile device 320, and may be output within an augmented reality environment provided by the augmented reality display module of the electronic device 310 in the present specification. The extended screen may include graphic representations of the mobile device 320, icons, content and application screens. For example, the extended screen may have a different resolution, different icon layout, and different ratio from the screen output from the display module of the mobile device 320 .

In the present specification, the augmented reality environment is a virtual space defined based on the electronic device 310 and may indicate an environment in which objects and/or virtual contents rendered on objects in the real world are overlapped and provided. Augmented reality may also be expressed as virtual reality (VR) and/or mixed reality (MR).

The virtual display manager 322 may manage application screens rendered on memory. For example, the aforementioned extended screen application 321 may create a virtual display, and the extended screen may be rendered on the virtual display. For reference, a plane on which a virtual display is disposed within an augmented reality environment (eg, a plane corresponding to an extended screen) may intersect with a direction of the user's gaze of the electronic device 310 .

The operating system 323 may manage execution and termination of each application, manager, and service of the mobile device 320 . The application manager 324 may manage priorities of applications (eg, application A and application B).

The video transmission service 326 may encode video information including video and extended screens, and transmit the encoded video information to other devices through a network. For example, the video transmission service 326 may transmit an extended screen.

The network interface 325 may manage data communication with other devices through a wireless communication module (eg, a UWB module and a Wi-Fi module).

The software architecture of the electronic device 310 may include a location service 311 , a target position manager 312 , an operating system 313 , a communication protocol 314 , and a network interface 315 .

The positioning service 311, in response to receiving a wireless communication signal (eg, a scan signal) from the external device 330, is associated with the external device 330 and the electronic device 310 that have transmitted the wireless communication signal. Information and attitude information can be calculated. The location information may include, for example, coordinates representing the location of the external device 330 based on the coordinate system of the electronic device 310 . The attitude information may include an angular difference between the orientation of the electronic device 310 and a direction from the external device 330 toward the electronic device 310 . The wireless communication signal may be a UWB scan signal transmitted in an ultra-wideband, but is not limited thereto. The orientation of the electronic device 310 is a direction in which one side of the device is viewed, and may be, for example, a direction perpendicular to the screen plane of the device. The direction of the electronic device 310 may indicate the same direction as the reference vector of the electronic device 310 .

The target position manager 312 calculates and manages location information for mapping and visualizing digital contents to objects and/or backgrounds in the real world within an augmented reality environment. can

The operating system 313 may manage execution and termination of each application, manager, and service of the electronic device 310 . Communication protocol 314 may represent a protocol for establishing communication with other devices. For example, the communication protocol 314 may establish communication with the external device 330 using a UWB protocol. The network interface 315 may perform data communication with the network interface 325 of another device.

The software architecture of the external device 330 may include a location service 331 , an operating system 333 , and a communication protocol 334 .

The positioning service 331 responds to receiving a wireless communication signal (eg, a scan signal) from the electronic device 310, and the external device 330 that has transmitted the wireless communication signal and location information related to the electronic device 310 and attitude information can be calculated. The location information may include, for example, a height difference between the external device 330 and the electronic device 310 based on the coordinate system of the electronic device 310 . The posture information may include, for example, an angular difference between the orientation of the external device 330 and a direction from the external device 330 toward the electronic device 310 .

The operating system 333 may manage execution and termination of each application, manager, and service of the external device 330 . Communication protocol 334 may represent a protocol for establishing communication with other devices. For example, the communication protocol 334 may establish communication with the electronic device 310 using a UWB protocol.

For reference, although the external device 330 is shown separately from the mobile device 320 in FIG. 3, it is not limited thereto. The external device 330 may be a mobile device 320 or a wireless communication device. For example, FIGS. 5 to 10 below describe examples in which the mobile device 320 operates as an external device 330 . When the external device 330 is the mobile device 320 , the mobile device 320 may provide the electronic device 310 with information about both the extended screen and the display reference position. As another example, FIGS. 11 to 15 illustrate an example in which a separate wireless communication device operates as an external device 330 . When the external device 330 is a wireless communication device, the mobile device 320 may provide an extended screen, and the wireless communication device may provide information about a display reference position. The wireless communication device may mainly refer to the electronic device 104 of FIG. 1 as an independent device separate from the mobile device 320 .

4 is a flowchart illustrating a method of providing an extended screen according to an exemplary embodiment.

In operation 410, an electronic device (eg, the wearable electronic device 200 of FIG. 2 or the electronic device 310 of FIG. 3 ) sends a mobile device (eg, the electronic device 101 of FIG. 1 or the mobile device of FIG. 3 ( 320)), a display reference position within the augmented reality environment to provide the extended screen may be determined based on wireless communication with an external device (eg, the external device 330 of FIG. 3 ). As described above with reference to FIG. 3 , the external device may be a mobile device or a separate wireless communication device. The wireless communication device may have a display module, but is not limited thereto, and may be a tag device lacking a display module as an example.

In operation 430, the electronic device provides at least some of the screens to the user through the augmented reality display module in response to a case where at least some of the extended screens to be provided at the display reference position are included in the viewing angle region of the augmented reality display module. can The viewing angle area of the augmented reality display module may indicate an area corresponding to an observable extent from the eye and/or eye box of a user wearing the electronic device. The screen display units 215a and 215b described above in FIG. 2 may cover at least a portion of the viewing angle region of the augmented reality display module, respectively. Virtual content and/or an extended screen may be displayed in an area covered by the screen display units 215a and 215b in the viewing angle area. The electronic device according to an embodiment may output a partial screen captured within the viewing angle area among the extended screens even before the display reference position within the viewing angle area is captured. The output of some screens will be described with reference to FIG. 16 below.

5 to 9 are diagrams illustrating examples of providing an extended screen based on a mobile device according to an exemplary embodiment.

5 illustrates an example in which the external device connected to the electronic device 510 is the mobile device 520 . For example, the mobile device 520 may execute the extended screen 590 application. The electronic device 510 may receive the extended screen 590 from the mobile device 520 . The electronic device 510 may output the aforementioned extended screen 590 on a virtual display plane determined based on the location of the mobile device 520 within the augmented reality environment 500 . The electronic device 510 may obtain location information and posture information between the mobile device 520 and the electronic device 510 while communicating with the mobile device 520 in a UWB band. The electronic device 510 and/or the mobile device 520 may estimate an angle of arrival (AOA) of the wireless communication signal by receiving the received wireless communication signal through multiple antennas. The electronic device 510 and/or the mobile device 520 may each determine attitude information (eg, an orientation of each device and an angular difference between directions in which two devices face each other) using the estimated AOA. Illustratively, the electronic device 510 is provided from the mobile device 520 within the augmented reality environment 500 from one surface on which the mobile device 520 is placed (eg, the ground or the surface of an object on which the mobile device 520 is placed). An extended screen 590 may be visualized at a spaced apart location and provided to the user. Estimation of location information and attitude information will be described with reference to FIGS. 6 to 9 below.

The electronic device 510 may change at least one of the position and posture of the plane on which the extended screen 590 is visualized in response to a change in at least one of the position and posture of the mobile device 520 .

6 to 9 are electronic devices when the external devices 602 and 720 (eg, the external device 330 of FIG. 3) are mobile devices (eg, the mobile device 320 of FIG. 3) according to an embodiment. An example of calculating location information and attitude information between 601 and 710 (eg, the electronic device 310 of FIG. 3 ) and the external device 602 and 720 will be described. 6 may show a flowchart illustrating calculation of location information and attitude information. FIG. 7 is a perspective view of the electronic devices 601 and 710 and the external devices 602 and 720 in an augmented reality environment, FIG. 8 is a top view, and FIG. 9 is a side view.

According to an embodiment, when the external devices 602 and 720 are mobile devices, the processors of the electronic devices 601 and 710 are spaced apart from one side of the mobile device (eg, the side where the display panel of the mobile device is disposed). may be determined as the display reference position 791 . The display reference position 791 is a position defined as a display standard of the extended screen 790 and may be a position where the center point of the extended screen 790 is disposed. For example, the electronic device 601 or 710 may determine the display reference position 791 based on the external device 602 or 720 through the operations illustrated in FIG. 6 . As shown in FIG. 7 , the first coordinate systems X1, Y1, Z1 of the electronic devices 601 and 710 and the second coordinate systems X2, Y2 and Z2 of the external devices 602 and 720 may be respectively defined. However, the electronic device 601 or 710 may calculate the coordinates of the display reference position 791 based on the first coordinate systems X1 , Y1 , and Z1 .

For example, in operation 611, the electronic devices 601 and 710 may transmit a wireless signal (eg, a wireless communication signal) for position calculation (eg, calculation of the aforementioned display reference position 791). For example, the electronic device 601 or 710 may transmit the above-described wireless communication signal to the external device 602 or 720 in a UWB band. However, the band of wireless communication is not limited to the UWB band, and communication may be performed in other communication bands. The electronic device 601 or 710 may radiate a wireless communication signal from the first reference point 711 .

In operation 612, the external device 602 or 720 may calculate a first angle 731 between a direction toward the electronic device 601 or 710 and a reference vector 722 of the external device 602 or 720. The attitude information between the external devices 602 and 720 and the electronic devices 601 and 710 may include a first angle 731 as a relative attitude between the electronic devices 601 and 710 and the external devices 602 and 720. The direction from the external device 602 or 720 toward the electronic device 601 or 710 is from the second reference point 721 of the external device 602 or 720 to the first reference point 711 of the electronic device 601 or 710. ) may be in the direction of The first reference point 711 may be, for example, a point where a wireless communication module of the electronic device 601 or 710 radiates a wireless communication signal, but is not limited thereto. The second reference point 721 may be a point where a wireless communication signal is radiated from a wireless communication module of the external device 602 or 720 by way of example, but is not limited thereto. The reference vector 722 of the external device 602 or 720 may be a vector perpendicular to the front surface of the external device 602 or 720 (eg, the surface on which the display panel of the mobile device is disposed) at the second reference point 721. there is. The external devices 602 and 720 may include a communication module having three or more antennas having different reception axes, and may calculate a first angle 731 based on AOAs receiving wireless communication signals from the antennas. .

In operation 613 , the external device 602 or 720 may calculate the height of the display reference position 791 separated from the external device 602 or 720 by a first distance 741 . According to an embodiment, the external device 602 or 720 may calculate a height coordinate spaced apart from one surface of the external device 602 or 720 by a first distance 741 . For example, the electronic device 601 or 710 and/or the external device 602 or 720 may obtain the display reference position 791 based on the user's gaze height. The aforementioned first distance 741 is a distance corresponding to a height from the ground and/or one surface of the external device 602 or 720, and may be determined based on the user's gaze height. However, it is not limited thereto, and the first distance 741 may be determined based on the height from the ground to the electronic device 601 or 710 without separately determining the eye height. As will be described later, the extended screen 790 can be provided at the user's eye level through the first distance 741 .

In addition, the electronic device 601 or 710 and/or the external device 602 or 720 determines the second distance 742 between the reference points 711 or 721 and the reference points relative to one side of the external device 602 or 720. When an angle formed by a straight line between (711, 721) is given, a trigonometric function (eg, a sine function) based on an angle formed by a straight line between the reference points 711 and 721 is applied to the second distance 742 to obtain a second distance. (741) can also be calculated. However, it is not limited thereto, and the second distance 741 may be a pre-designated value.

In operation 614, the external devices 602 and 720 may transmit the first angle 731 and the height of the display reference position 791 through wireless communication.

In operation 615, the electronic device 601 or 710 may calculate a second angle 732 and a second distance 742 based on the wireless signal received from the external device 602 or 720. For example, the external devices 602 and 720 may transmit a wireless communication signal together with the height of the first angle 731 and the display reference position 791 in operation 614 described above. The wireless communication module of the electronic device 601 or 710 may include three or more antennas having different reception axes, and the electronic device 601 or 710 may receive a control signal based on the AOA received through the antennas. 2 angles 732 can be calculated. The electronic devices 601 and 710 illustratively provide a second signal between the reference points 711 and 721 based on the reception strength of the wireless communication signal and/or the time-of-flight (TOF) required for signal transmission. Distance 742 can be calculated.

In operation 616, the electronic device 601 or 710 sets a first reference point of the electronic device 601 or 710 in a direction forming a second angle 732 with respect to the reference vector 712 of the electronic device 601 or 710. Plane coordinates of the second reference point 721 of the external devices 602 and 720 spaced apart from 711 by the second distance 742 may be calculated based on the first coordinate system of the electronic devices 601 and 710 .

In operation 617 , the electronic devices 601 and 710 may determine the height of the display reference position 791 and the plane coordinates of the second reference point as the display reference position 791 . In other words, the electronic device 601 or 710 may determine the height coordinates obtained in operation 613 and the plane coordinates obtained in operation 617 as coordinates of the display reference position 791 . The display reference location 791 may be a floating location from the external devices 602 and 720 .

In operation 620 , the electronic device 601 or 710 may rotate the plane 792 of the extended screen 790 based on the first angle 731 and the second angle 732 . For example, the electronic device 601 or 710 may determine the rotation angle of the plane 792 of the extended screen 790 by subtracting the second angle from the first angle. Therefore, when the mobile device is rotated based on an axis perpendicular to one surface of the mobile device, the processor of the electronic device 601 or 710 moves the plane ( 792) can be rotated. Accordingly, the plane 792 of the extended screen 790 may be rotated based on a rotational axis parallel to the reference axis of the external devices 602 and 720 (eg, a central axis perpendicular to one surface on which the display panel is disposed). When the external devices 602 and 720 are rotated clockwise with respect to the reference axis, the plane 792 of the extended screen 790 may be rotated clockwise with respect to the rotation axis and output in operation 633 described later. . Similarly, when the external devices 602 and 720 are rotated counterclockwise with respect to the reference axis, the plane 792 of the extended screen 790 is rotated counterclockwise with respect to the rotation axis in operation 633 described below. and can be output.

In operation 631, the electronic device 601 or 710 may transmit a display preparation completion event of the extended screen 790.

In operation 632, the external devices 602 and 720 may transmit an extended screen 790. For example, the external device 602 or 720 may transmit an extended screen 790 to the electronic device 601 or 710 in response to receiving a display preparation completion event from the electronic device 601 or 710 .

In operation 633, the electronic devices 601 and 710 may output the extended screen 790 at the display reference position 791. For example, the electronic device 601 or 710 responds to receiving the extended screen 790 from the external device 602 or 720 on the virtual display plane 792 determined based on the display reference position 791. An extended screen 790 may be provided. When the rotation angle is determined in operation 620, the electronic device 601 or 710 may output an extended screen 790 rotated by the rotation angle. The processor of the electronic device 601 or 710 displays the extended screen 790 to the user on a plane that is non-parallel to the screen plane of the mobile device (eg, one side on which the display panel is disposed) in an augmented reality environment. can provide For example, the electronic devices 601 and 710 may output the extended screen 790 on a virtual display plane substantially perpendicular to the screen plane.

For reference, operations 615, 616, and 617 in FIG. 6 are described as being performed by the electronic device 601 and 710, and operations 612, 613, and 614 performed by the external device 602 and 720. , but is not limited thereto. 6 is purely an example, and operations 612 to 617 may be performed by one of the electronic devices 601 and 710 and the external devices 602 and 720 . In addition, the operation sequence is not limited to that shown in FIG. 6 . For example, operations 612, 613, and 614 may be performed later than operations 615 and 616. In this case, in operation 613, the height of the display reference position 791 may be calculated using the second angle and the second distance.

10 illustrates an example of providing an extended screen when a mobile device is hidden according to an embodiment.

The electronic device 1010 according to an embodiment may establish communication with an external device 1020 (eg, a mobile device) even when the external device 1020 is not visually exposed. For example, the electronic device 1010 searches for the external device 1020 covered by the object 1050 using a wireless communication signal (eg, a UWB band signal) within a space corresponding to the augmented reality environment 1000. can do. 10 shows an example in which the external device 1020 is stored in a desk drawer.

When the electronic device 1010 detects the external device 1020 in a space corresponding to the augmented reality environment 1000, the electronic device 1010 has a field of view (eg, a field of view of the camera 245 for photographing in FIG. 2). At this point, it is possible to visually identify the external device 1020. The electronic device 1010 may visualize notification content indicating a hidden location of the external device 1020 in response to a case where the external device 1020 is not visually identified within the viewing angle. For example, the electronic device 1010 may visualize notification content at a point 1025 on an object 1050 that covers the external device 1020 . The notification content may include a graphic representation explicitly or implicitly indicating the type of external device 1020 .

In addition, the processor of the electronic device 1010 determines a point 1025 on the object 1050 in response to a case where the external device 1020 is covered by the object 1050 between the external device 1020 and the electronic device 1010. It is possible to determine the display reference position based on . The electronic device 1010 determines a position spaced apart from the point 1025 on the object 1050 in a direction perpendicular to one of the ground, one surface of the object 1050, and one surface of the external device 1020 as the display reference position. can The electronic device 1010 may visualize the extended screen 1090 of the mobile device at the determined display reference position. While visualizing the extended screen 1090, the electronic device 1010 sets the coordinates of the display reference position in response to the relative position change and angle change between the electronic device 1010 and the external device 1020 of the wireless communication signal described above. It can be updated in real time using AOA calculation according to transmission and reception. Therefore, even if the external device 1020 is covered by an object 1050, the electronic device 1010 uses a wireless communication signal (eg, a UWB band signal) to display the extended screen 1090 at a location adjacent to the external device 1020. ) can be displayed. The electronic device 1010 may provide the extended screen 1090 in a more intuitive location.

In addition, the notification content may include information representing messages received from the external device 1020 (eg, the number of unread messages after reception and a summary of received message contents). The electronic device 1010 may visualize the aforementioned notification content at a location corresponding to the external device 1020 in response to receiving a message from the external device 1020 . The electronic device 1010 may visualize the extended screen 1090 in response to a manipulation of activating the notification content (eg, a manipulation of the user's body to contact a space occupied by the notification content within the augmented reality environment 1000). . Thus, users can easily view notifications and handle related tasks with the desktop experience.

11 to 15 are diagrams illustrating examples of providing an extended screen based on a wireless communication device according to an embodiment.

11 illustrates an example in which the external device 1120 connected to the electronic device 1110 is a separate and independent wireless communication device from the mobile device 1130 . In FIG. 11, the wireless communication device is illustrated as a device including a display module, but this is purely an example and is not limited thereto.

According to an embodiment, the mobile device 1130 may execute an extended screen application. The electronic device 1110 may receive the extended screen 1190 from the mobile device 1130 . The electronic device 1110 may output the aforementioned extended screen 1190 on a virtual display plane determined based on the location of the external device 1120 within the augmented reality environment 1100 .

The electronic device 1110 may acquire location information and attitude information between the external device 1120 and the electronic device 1110 while performing communication with the external device 1120 in a UWB band. The electronic device 1110 and/or the external device 1120 may estimate an angle of arrival (AOA) of the wireless communication signal by receiving the received wireless communication signal through multiple antennas. The electronic device 1110 and/or the external device 1120 may determine attitude information (eg, the orientation of each device and the angle difference between the directions in which the two devices face each other) using the estimated AOA. For example, the electronic device 1110 may visualize an extended screen 1190 at a location spaced apart from the external device 1120 within the augmented reality environment 1100 and provide it to the user. For example, the electronic device 1110 may determine a position where the display module of the external device 1120 and the extended screen 1190 do not overlap as the display reference position. The electronic device 1110 may output an extended screen to the periphery of the display module of the external device 1120 (eg, one side of the side, top, and bottom, excluding the front and rear of the display panel). Accordingly, the electronic device 1110 may provide the extended screen 1190 to the user without being disturbed by the screen of the display module of the external device 1120 . To this end, the external device 1120 provides physical size information (eg, screen size, location of the communication module within the device) of the display module of the external device 1120 to the electronic device 1110 during wireless communication (eg, UWB band communication). ) can be shared. The electronic device 1110 and/or the external device 1120 may calculate an accurate display reference position using the screen size of the display module of the shared external device 1120 . Estimation of location information and attitude information will be described with reference to FIGS. 12 to 14 below.

The electronic device 1110 may change at least one of the position and attitude of the plane on which the extended screen 1190 is visualized in response to a change in at least one of the position and attitude of the external device 1120 . In other words, the electronic device 1120 may move and rotate the extended screen in response to a user's manipulation (eg, movement and rotation) of the external device 1120 . For example, the electronic device 1120 may move the extended screen 1190 in response to a change in location of the external device 1120 in response to the movement of the external device 1120 .

12 to 15 illustrate electronic devices 1201 and 1310 (eg, the external device 330 of FIG. 3 ) when the external devices 1202 and 1320 (eg, the external device 330 of FIG. 3 ) according to an embodiment are wireless communication devices. An example of calculating location information and posture information between the electronic device 310 and an external device will be described. 12 may show a flowchart illustrating calculation of location information and attitude information. 13 is a perspective view of the electronic devices 1201 and 1310 and external devices 1202 and 1320 in an augmented reality environment, FIG. 14 is a top view when stationary, and FIG. 15 is a top view when rotated. Although a mobile device (eg, the mobile device 320 of FIG. 3 ) is not shown in FIGS. 13 to 15 , the mobile device may establish communication with the external devices 1202 and 1320 and the electronic devices 1201 and 1320 . .

According to an embodiment, when the external devices 1202 and 1320 are other wireless communication devices, the processor of the electronic device 1201 or 1310 displays a location designated by the wireless communication device relative to the wireless communication device as a reference location 1391 can be determined by The display reference position 1391 is a position defined as a display standard of the extended screen 1390 and may be a position where the center point of the extended screen 1390 is disposed. For example, the electronic device 1201 or 1310 may determine the display reference position 1391 based on the external device 1202 or 1320 through the operations illustrated in FIG. 12 . As shown in FIG. 13 , the first coordinate systems X1, Y1, Z1 of the electronic devices 1201 and 1310 and the second coordinate systems X2, Y2 and Z2 of the external devices 1202 and 1320 may be respectively defined. However, the electronic devices 1201 and 1310 may calculate the coordinates of the display reference position 1391 based on the first coordinate system X1 , Y1 , and Z1 .

For example, in operation 1211, the electronic devices 1201 and 1310 may transmit a wireless signal (eg, a wireless communication signal) for position calculation (eg, calculation of the aforementioned display reference position 1391). For example, the electronic device 1201 or 1310 may transmit the aforementioned wireless communication signal to the external device 1202 or 1320 in the UWB band. However, the band of wireless communication is not limited to the UWB band, and communication may be performed in other communication bands. The electronic device 1201 or 1310 may radiate a wireless communication signal from the first reference point 711 .

In operation 1212, the external device 1202 or 1320 sets a first angle ( 1331) can be calculated. The attitude information between the external devices 1202 and 1320 and the electronic devices 1201 and 1310 may include a first angle 1331 as a relative attitude between the electronic devices 1201 and 1310 and the external devices 1202 and 1320. The direction from the external device 1202 or 1320 toward the electronic device 1201 or 1310 is from the second reference point 1321 of the external device 1202 or 1320 to the first reference point 1311 of the electronic device 1201 or 1310. ) may be in the direction of The first reference point 1311 may be, for example, a point from which wireless communication modules of the electronic devices 1201 and 1310 emit wireless communication signals, but is not limited thereto. The second reference point 1321 may be a point where a wireless communication signal is radiated from the wireless communication module of the external device 1202 or 1320 by way of example, but is not limited thereto. The reference vector 1322 of the external devices 1202 and 1320 is a vector perpendicular to the front surface of the external devices 1202 and 1320 (eg, the side on which the display panel of the wireless communication device is disposed) at the second reference point 1321. can The external devices 1202 and 1320 may include a communication module having three or more antennas having different reception axes, and may calculate a first angle 1331 based on AOAs receiving wireless communication signals from the antennas. .

In operation 1213, the external device 1202 or 1320 may transmit the size and first angle 1331 of the display module of the wireless communication device to the electronic device 1201 or 1310. Information about the size of the display module may include, for example, one or a combination of two or more of a diagonal length of a plane, a horizontal length of a plane, and a vertical length of a plane of the display panel.

However, the external devices 1202 and 1320 are not limited to transmitting the screen size of the display module. For example, the external devices 1202 and 1320 transmit information defining the relative position of the display reference position 1391 relative to the second reference point 1321 of the corresponding device (eg, relative distance and relative direction) to the electronic device ( 1201, 1310). Accordingly, the location where the extended screen is displayed may be indicated by the external devices 1202 and 1320 .

In operation 1214, the electronic device 1201 or 1310 may calculate a second angle 1332 and a second distance 1342 based on the wireless signal received from the external device 1202 or 1320. For example, the external devices 1202 and 1320 may transmit a wireless communication signal together with the first angle 1331 and the screen size of the display module in the above-described operation 1213 . The wireless communication module of the electronic device 1201 or 1310 may include three or more antennas having different reception axes, and the electronic device 1201 or 1310 may receive a control signal based on the AOA received through the antennas. 2 angles 1332 can be calculated. The electronic devices 1201 and 1310 illustratively perform a second operation between the reference points 1311 and 1321 based on the reception strength of the wireless communication signal and/or the time-of-flight (TOF) required for signal transmission. Distance 1342 can be calculated.

In operation 1215, the electronic device 1201 or 1310 sets a first reference point of the electronic device 1201 or 1310 in a direction forming a second angle 1332 with respect to the reference vector 1312 of the electronic device 1201 or 1310. Plane coordinates of the second reference point 1321 of the external devices 1202 and 1320 spaced apart from 1311 by the second distance 1342 may be calculated based on the first coordinate system of the electronic device 1201 and 1310.

In operation 1216, the electronic devices 1201 and 1310 may calculate a display reference position 1391 separated from the second reference point 1321 by a first distance 1341 using the screen size of the wireless communication device. For example, the electronic device 1201 or 1310 may determine a position separated by a distance determined based on the screen size of the display module as the display reference position 1391 in response to a case where the wireless communication device includes the display module. there is. In other words, the electronic devices 1201 and 1310 may determine the first distance 1341 based on the screen size of the display module. The first distance 1341 is a non-overlapping distance between the display screen and the extended screen of the display module, and is, for example, longer than or equal to one of a horizontal length, a vertical length, and a diagonal length of the display module. can be When the extended screens 1390 are arranged horizontally with respect to the wireless communication device, the first distance 1341 may have a value greater than or equal to the horizontal length of the display module of the wireless communication device. For another example, when the extended screen 1390 is arranged vertically with respect to the wireless communication device, the first distance 1341 may have a value greater than or equal to the vertical length of the display module of the wireless communication device. As another example, when the extended screens 1390 are arranged diagonally with respect to the wireless communication device, the first distance 1341 may have a value greater than or equal to the diagonal length of the display module of the wireless communication device. Accordingly, the display reference position 1391 may be a spaced position so that the extended screen 1390 does not interfere with the screen output of the external devices 1202 and 1320 .

In operation 1220, the electronic devices 1201 and 1310 may rotate the plane of the extended screen 1390 based on the first angle 1331 and the second angle 1332. For example, the electronic device 1201 or 1310 may determine the rotation angle of the plane of the extended screen 1390 by subtracting the second angle 1332 from the first angle 1331 . Therefore, the processors of the electronic devices 1201 and 1310, when the display module is rotated based on the rotational axis of the wireless communication device, extends the screen 1390 in a direction different from the rotational direction of the display module about the rotational axis (eg, the opposite direction). can rotate the plane of When the external devices 1302 and 1320 are rotated clockwise with respect to the reference axis, the plane of the extended screen 1390 may be rotated counterclockwise with respect to the rotation axis in operation 1233 to be described later and output. Similarly, when the external devices 1202 and 1320 are rotated counterclockwise with respect to the reference axis, the plane of the extended screen 1390 may be rotated clockwise with respect to the rotation axis and outputted in operation 1233 described later. there is.

In operation 1231, the electronic devices 1201 and 1310 may transmit a display preparation completion event of the extended screen 1390 to the mobile device 1203.

In operation 1232, the mobile device 1203 may transmit an extended screen 1390. For example, the mobile device 1203 may transmit the extended screen 1390 to the electronic device 1201 or 1310 in response to receiving a display preparation completion event from the electronic device 1201 or 1310 .

In operation 1333 , the electronic devices 1201 and 1310 may output the extended screen 1390 at the display reference position 1391 . For example, in response to receiving the extended screen 1390 from the mobile device 1203, the electronic devices 1201 and 1310 display the extended screen 1390 on the virtual display plane determined based on the display reference position 1391. ) can be provided. When the rotation angle is determined in operation 1220, the electronic device 1201 or 1310 may output the extended screen 1390 rotated by the rotation angle. For reference, the angle formed by the screen plane of the display module of the wireless communication device with respect to the ground within the augmented reality environment may be the same as the angle formed by the plane of the extended screen 1390 with respect to the ground. Accordingly, the electronic devices 1201 and 1310 may provide a more consistent desktop experience to the user by providing an extended screen having the same tilting angle as that of the display module of the wireless communication device without a separate setting.

For reference, although operations 1214, 1215, and 1216 are performed by the electronic device 1201 and 1310 in FIG. 12 and operations 1212 and 1213 are performed by the external device 1202 and 1320, It is not limited to this. 12 is purely an example, and operations 1212 to 1216 may be performed by one of the electronic devices 1201 and 1310 and the external devices 1202 and 1320 . In addition, the operation sequence is not limited to that shown in FIG. 12 . For example, operations 1212 and 1213 may be performed after operations 1214, 1215 and 1216.

16 is a diagram for explaining provision of a part of an extended screen according to a viewing angle of an electronic device according to an exemplary embodiment.

As described above, the electronic device 1610 according to an embodiment may establish a connection with an external device (eg, a mobile device or a wireless communication device) through wireless communication (eg, UWB band communication). The electronic device 1610 may wirelessly establish communication even when an external device is not visually observed within an area corresponding to the viewing angle. The electronic device 1610 may perform a preparation operation for visualizing the extended screen 1690 of the mobile device in response to establishing wireless communication with a non-observable external device out of the viewing angle of the corresponding device. can After establishing a connection with the external device, the electronic device 1610 may calculate and track a positional relationship between an area corresponding to the viewing angle of the electronic device 1610 and the extended screen 1690 . For example, the electronic device 1610 displays an extended screen 1690 based on a display reference position based on a first reference point (eg, a point where a wireless communication module of the electronic device 1610 radiates a wireless communication signal). The plane equation to be placed can be determined. The electronic device 1610 may calculate two linear equations representing the boundary line of the viewing angle in a plane (eg, X1Y1 plane of the first coordinate system of the electronic device 1610) based on the first reference point. The electronic device 1610 may determine an intersection point between the two linear equations and the aforementioned plane equation. The electronic device 1610 may monitor whether the intersection point between the two linear equations and the plane equation is within the plane area of the extended screen 1690 determined based on the display reference position.

The electronic device 1610 responds to a case where at least a portion of the screen 1695 of the extended screen 1690 to be provided at the display reference position is included within a field of view (FOV) area of the augmented reality display module, at least some of the screen. (1695) may be provided to the user through the augmented reality display module. For example, when the electronic device 1610 detects an intersection point between the above-mentioned two straight line equations and a plane equation within a plane area of the extended screen 1690, the electronic device ( A partial area 1695 included in the viewing angle of 1610 may be determined. In other words, even if the electronic device 1610 detects an external device in a wireless communication method, if the extended screen 1690 is not included within the viewing angle, the output is held, and if the partial area 1695 comes within the viewing angle, the partial area 1695 is displayed. can be provided to users. The electronic device 1610 may provide the entire area of the extended screen 1690 within the augmented reality environment when the entire extended screen 1690 is within the viewing angle. Since the electronic device 1610 does not require vision recognition of the marker, as described above, the electronic device 1610 may immediately provide the partial screen 1695 to the user.

Also, the communication module of the electronic device 1610 may perform communication with an external device at a first data rate when at least some screens are not included in the viewing angle area. The electronic device 1610 may communicate with an external device at a second data rate different from the first data rate when at least a portion of the screen is included in the viewing angle area. The first data rate may be lower than the second data rate. In other words, the electronic device 1610 is positioned between the viewing angle of the electronic device 1610 and the extended screen according to the display reference position determined based on the reference point of the external device at a low data rate until the extended screen 1690 enters the viewing angle. Relationships can be monitored. The electronic device 1610 may minimize disconnection in reproduction of the extended screen 1690 by increasing the communication speed at the second data rate when some of the screens 1695 of the extended screen 1690 come within the viewing angle. The second data rate is illustratively determined based on a frame rate (eg, 60 frames per second (FPS), and 120 FPS) for providing a screen provided from the augmented reality display module of the electronic device 1610. can

17 is a diagram illustrating rotation tracking using an inertial sensor according to an exemplary embodiment.

The processor of the electronic device 1710 according to an embodiment, in response to a case where rotation detection using a wireless communication signal is unavailable, based on at least one of rotation information and movement information detected using an inertial sensor in an external device, At least one of rotation and movement of the extended screen may be performed. For example, the electronic device 1710 automatically performs a location tracking function using an inertial sensor of an external device (e.g., in response to failure in receiving a wireless communication signal (eg, a UWB signal) depending on the surrounding environment). IMU (inertial measurement unit) mode) can be activated. However, it is not limited thereto, and the electronic device 1710 may initiate location tracking using an inertial sensor in response to a manual user input. For example, the external device may track a change in the first angle 1731 using an inertial sensor, and transmit the tracked amount of change in the first angle 1731 to the electronic device 1710 . The electronic device 1710 may track the amount of change in the second angle according to its own rotation by using its own inertial sensor. The electronic device 1710 may rotate the extended screen based on the variation of the first angle 1731 and the second angle individually tracked using an inertial sensor.

In FIG. 17 , rotation tracking using an inertial sensor has been illustratively described, but is not limited thereto. The electronic device 1710 and/or an external device may also track the positional movement of each device using an inertial sensor and share the same with other devices.

18 to 21 are views illustrating control of an extended screen based on manipulation of a mobile device according to an exemplary embodiment.

According to an embodiment, a processor of an electronic device may control an extended screen based on a gesture input to the mobile device.

For example, in operation 1841, the electronic device may identify an alignment direction of the mobile device with respect to the electronic device. The electronic device may determine whether the mobile device is horizontally aligned or vertically aligned. The processor of the electronic device may provide the extended screen as one of the horizontal screen 2192 and the vertical screen 2191 based on alignment between the electronic device and the mobile device. The electronic device may determine that the mobile device is vertically aligned with respect to the electronic device in response to a difference in direction between a reference vector of the electronic device and a longitudinal axis of the mobile device (eg, a longitudinal axis of the mobile device) being less than a threshold difference. can Conversely, the electronic device may determine that the mobile device is horizontally aligned with respect to the electronic device in response to a case where a difference in direction between the reference vector of the electronic device and the vertical axis of the mobile device is greater than or equal to a critical difference. For reference, as in the example described in FIG. 17 , when the electronic device and the mobile device initiate rotation tracking based on the inertial sensor, when the mobile device rotates at a threshold angle or more based on the reference vector of the mobile device, the electronic device It may be determined that the alignment of the mobile device is switched (eg, switching from horizontal alignment to vertical alignment or switching from vertical alignment to horizontal alignment) ( 2103 ).

In operation 1842, the electronic device may output the extended screen to the vertical screen 2191 when the mobile device is vertically aligned. The vertical screen 2191 may represent a screen in which the vertical length of the screen is greater than the horizontal length. In operation 1843, the electronic device may output the extended screen to the horizontal screen 2192 when the mobile device is horizontally aligned. The horizontal screen 2192 may represent a screen in which the horizontal length of the screen is greater than the vertical length.

However, it is not limited to this, and the electronic device outputs the extended screen to the horizontal screen 2192 when the horizontal screen is output from the mobile device, and outputs the extended screen to the vertical screen 2191 when the vertical screen is output from the mobile device. You may.

In operation 1844, the external device may transmit the extended screen to the electronic device. For example, the external device may transmit the extended screen determined by the above-described operations 1841, 1842, and 1843 to the electronic device.

In operation 1851, the electronic device may monitor whether a first gesture 1901 is generated with respect to the mobile device. In operation 1852, the electronic device may change the size of the extended screen 1990 in response to detecting the first gesture 1901. For example, the processor of the electronic device may change the size of the extended screen 1990 in response to a first gesture 1901 input to the mobile device. As shown in FIG. 19 , the electronic device responds to a first gesture 1901 in which a distance between two touch points on the screen of the mobile device changes (eg, pinch zoom-in and zoom-out operations) to expand the size of the screen 1990. can be changed. For example, the electronic device may decrease the size of the extended screen 1990 when the distance between the two touch points decreases, and increase the size of the extended screen 1990 when the distance between the two touch points increases. .

In operation 1853, the electronic device may monitor whether the second gesture 2002 has occurred with respect to the mobile device. In operation 1854, the electronic device may change the brightness of the extended screen 2090 in response to detecting the second gesture 2002. For example, the processor of the electronic device may change the brightness of the extended screen 2090 in response to the input of the second gesture 2002 to the mobile device. As shown in FIG. 20 , the electronic device may change the brightness of the extended screen 2090 in response to a linear movement along one axis of a touch input to the screen of the mobile device (eg, a scroll input). For example, the electronic device increases the brightness of the extended screen 2090 when the touch point moves in a first direction along one axis, and increases the brightness of the extended screen 2090 when the touch point moves in a second direction opposite to the first direction. ) can reduce the brightness.

The above-described gesture inputs are detected by the mobile device itself, and the mobile device may request an operation (eg, screen size change or brightness change) according to each gesture to the electronic device, but is not limited thereto. The above-described gesture inputs may be visually recognized by an electronic device.

Claims (20)

In electronic devices,
A wireless communication module for wireless communication with an external device;
An augmented reality display module that outputs an extended screen of a mobile device;
a memory in which computer-executable instructions are stored; and
A processor that accesses the memory and executes the instructions
including,
These commands are
Determines a display reference position within an augmented reality environment to provide the extended screen received from the mobile device based on wireless communication with the external device;
In response to a case where at least some screens of the extended screen to be provided at the display reference position are included in a field of view (FOV) area of the augmented reality display module, the at least some screens are displayed through the augmented reality display module. set to provide to the user,
electronic device.
According to claim 1,
the processor,
When the external device is the mobile device, acquiring a location spaced apart from one surface of the mobile device as a display reference location.
electronic device. According to claim 2
the processor,
Providing the extended screen to the user on a plane that is non-parallel to the screen plane of the mobile device in the augmented reality environment,
electronic device. According to claim 2
the processor,
When the mobile device is rotated based on one axis perpendicular to one surface of the mobile device, rotating the plane of the extended screen in the same direction as the rotation direction of the mobile device about the one axis,
electronic device. According to claim 2,
the processor,
determining the display reference position based on the user's gaze height;
electronic device. According to claim 1,
the processor,
When the external device is another wireless communication device, determining a position designated by the wireless communication device with respect to the wireless communication device as a display reference position,
electronic device. According to claim 6,
the processor,
In response to the case where the wireless communication device includes a display module, determining a position spaced apart by a distance determined based on a screen size of the display module as the display reference position,
electronic device. According to claim 6,
The angle formed by the screen plane of the display module of the wireless communication device with respect to the ground in the augmented reality environment is the same as the angle formed by the plane of the extended screen with respect to the ground,
electronic device. According to claim 6,
the processor,
When the display module is rotated based on the rotational axis of the wireless communication device, rotating the plane of the extended screen in a direction different from the rotational direction of the display module about the rotational axis,
electronic device. According to claim 1,
the processor,
Moving the extended screen in response to a change in location of the external device in response to the movement of the external device.
electronic device. According to claim 1,
The wireless communication module,
Including three or more antennas having different receiving axes,
electronic device. According to claim 1,
The external device includes a communication module having three or more antennas having different receiving axes,
electronic device. According to claim 1,
the processor,
Performing at least one of rotation and movement of the expanded screen based on at least one of rotation information and movement information detected using an inertial sensor in the external device in response to a case where rotation detection using a wireless communication signal is unavailable ,
electronic device. According to claim 1,
the processor,
Controlling an extended screen based on a gesture input to the mobile device,
electronic device. According to claim 14,
the processor,
In response to a first gesture input to the mobile device, changing the size of the extended screen;
Changing the brightness of the extended screen in response to a second gesture input to the mobile device,
electronic device. According to claim 1,
the processor,
Providing the extended screen as one of a horizontal screen and a vertical screen based on alignment between the electronic device and the mobile device,
electronic device. According to claim 1,
The communication module,
Performing communication with the external device at a first data rate when the at least part of the screen is not included in the viewing angle area;
performing communication with the external device at a second data rate different from the first data rate when the at least part of the screen is included in the viewing angle area;
electronic device. According to claim 1,
the processor,
determining the display reference position based on a point on the object in response to a case where the external device is covered by an object between the external device and the electronic device;
electronic device.
In a computer-implemented method,
determining a display reference position to provide an extended screen of the mobile device within an augmented reality environment based on wireless communication with an external device; and
In response to a case where at least some of the extended screens to be provided at the display reference position are included in the field of view (FOV) area of the augmented reality display module, the at least some of the screens are displayed to the user through the augmented reality display module. action to provide
How to include. A computer readable recording medium storing one or more computer programs including instructions for performing the method of claim 19 .

Images