KR20240043034A - Apparatus for expanding screen in an electronic device and operating method therefor - Google Patents

Abstract

Embodiments of the present disclosure relate to an electronic device that displays a mirroring screen by linking with an external electronic device and a method of operating the same. For this purpose, the external device can acquire mirroring data and display a mirrored image on the first display area allocated on the display using the acquired mirroring data. The external device obtains extended mirroring data from previous mirroring data that has been used to display the mirrored image, and displays at least one extended display additionally allocated in addition to the first display area on the display according to a user's gesture. An extended mirroring image can be displayed in the area using the extended mirroring data.

Description

Screen expansion device and operating method thereof in electronic device {APPARATUS FOR EXPANDING SCREEN IN AN ELECTRONIC DEVICE AND OPERATING METHOD THEREFOR}

Embodiments of the present disclosure relate to a screen expansion device and a method of operating the same in an electronic device.

Digital devices can install various applications (hereinafter referred to as “apps”) that provide specific functions or services. The digital device may execute an installed app in response to a user's request. The digital device can display information according to the execution of the app through a display. The digital device may be an electronic device that is easy for a user to carry, such as a smart phone, mobile phone, or tablet device.

Efforts are being made to reduce the size of the portable digital device so that it can be designed or easily held by the user. Because of this, digital devices may have a relatively small screen size compared to display devices such as televisions or monitors. A digital device with a relatively small screen size compared to the display device may halve the user's satisfaction when using multimedia content that provides moving images, such as movies, dramas, or games.

An embodiment of the present disclosure can provide an electronic device that expands the mirroring screen through interaction with a user in a mirroring system that mirrors the screen of a digital device, and a method of operating the same.

According to one embodiment, a device includes a display module configured to display a mirrored image on a display window, a communication module configured to communicate with an external device, detecting a user's gesture, and performing sensing corresponding to the detected user's gesture. A sensor module configured to generate a signal, a memory configured to be allocated one or a plurality of buffer areas, and electrically connected to the display module, the communication module, or the memory, and perform the operation of the display module, the communication module, or the memory. At least one processor configured to control, wherein the at least one processor receives mirroring data from the external device through the communication module, and uses the mirroring data to perform mirroring on a first display area allocated in the display window. Controlling the display module to display an image, and performing extended mirroring using extended mirroring data in at least one extended display area additionally allocated in addition to the first display area in the display window according to a user gesture detected by the sensor module. and configured to control the display module to display an image, wherein the extended mirroring data can be obtained from previous mirroring data that has been used to display the mirrored image.

According to one embodiment, a display method in an electronic device may include an operation of acquiring mirroring data. The display method may include displaying a mirrored image on a first display area allocated on a display using the obtained mirroring data. The display method may include obtaining extended mirroring data from previous mirroring data that has been used to display the mirrored image. The display method may include displaying an extended mirroring image using the extended mirroring data in at least one additionally allocated extended display area in addition to the first display area on the display according to a user's gesture.

According to an embodiment of the present disclosure, an external device that provides a mirroring screen recycles the mirroring image that was provided as a mirroring screen through interaction with the user to provide an extended mirroring screen, thereby reducing traffic or power consumption of the wireless channel. there is.

The technical problems to be achieved in the present disclosure are not limited to the above-mentioned technical problems, and other technical problems not mentioned above may be derived from the exemplary embodiments of the present disclosure by those skilled in the art. there is.

Effects that can be obtained from the exemplary embodiments of the present disclosure can be clearly derived and understood by those skilled in the art from the following description. That is, unintended effects resulting from implementing the exemplary embodiments of the present disclosure may also be derived by those skilled in the art from the exemplary embodiments of the present disclosure.

FIG. 1 is a diagram conceptually illustrating a mirroring function performed in a mirroring system according to an embodiment.
FIG. 2 is a diagram illustrating a procedure for a mirroring function in a mirroring system, according to an embodiment.
Figure 3 is a configuration diagram of a mirroring system for supporting screen mirroring, according to an embodiment.
FIG. 4 is a diagram illustrating the functional configuration of a source device supporting screen mirroring, according to an embodiment.
FIG. 5 is a diagram illustrating the functional configuration of an external device supporting screen mirroring, according to an embodiment.
Figure 6 is an example of a screen display of a virtual space on an external device, according to an embodiment.
Figure 7 is an example of expanding a mirroring screen on an external device of a mirroring system, according to an embodiment.
FIG. 8 is a diagram illustrating a processing procedure for providing a mirroring function in a mirroring system, according to an embodiment.
FIG. 9 is a diagram illustrating processing procedures for mirroring and event processing operations in a mirroring system, according to an embodiment.
10A and 10B are control flowcharts for an external device to display a mirrored image in a mirroring system, according to an embodiment.
FIG. 11A is an example diagram of expanding a mirroring screen in an external device that supports a mirroring function, according to an embodiment.
FIG. 11B is an example diagram of expanding a mirroring screen in an external device that supports a mirroring function, according to an embodiment.
Figure 12 is an example diagram of expanding a mirroring screen using a 3D mirroring image in a mirroring system, according to an embodiment.
FIG. 13 is an example diagram showing interaction in an extended display area from an external device constituting a mirroring system, according to an embodiment.
Figure 14 is a history hierarchy diagram for managing an extended display area in a mirroring system, according to an embodiment.
FIG. 15 is an example diagram of independently operating an extended display area in a mirroring system, according to an embodiment.
FIG. 16 is a diagram illustrating an example of expanding a mirroring image in time and space in an external device, according to an embodiment.
FIG. 17 is a control flowchart for expanding a mirroring image in time and space in an external device, according to an embodiment.
18 is a block diagram of an electronic device in a network environment, according to various embodiments.
In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components.

Hereinafter, with reference to the drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily practice them. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components. Additionally, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

FIG. 1 is a diagram conceptually illustrating a mirroring function performed by the mirroring system 1 according to an embodiment.

Referring to FIG. 1, the mirroring system 1 may include at least two electronic devices. The at least two electronic devices may include a source device 10, which is a first electronic device, and an external device 20, which is at least one second electronic device.

The source device 10 may generate a mirrored image according to the mirroring function and provide the mirrored image to the external device 20. For example, the source device 10 creates a virtual display in response to an app execution request from the external device 20, executes the target app on the virtual display, and mirrors it to the external device 20. An image (hereinafter referred to as a 'mirroring image') can be acquired. For example, the external device 20 may include a mirroring image in mirroring data and transmit it to the external device 20. The mirroring data may include data to be used to mirror the mirroring image (hereinafter referred to as 'mirroring metadata'). The mirroring metadata may include, for example, information such as size, resolution, DPI, or display position of the mirrored image. The source device 10 may be, for example, a digital device such as a smartphone or tablet PC on which a program such as an app can be installed or executed.

The external device 20 may be a screen expansion device that expands the area to display the mirrored image provided by the source device 10 (hereinafter referred to as 'mirroring display area') for the mirroring function. The external device 20 may be, for example, a digital device such as a wearable device with its own display, a smart phone, a tablet PC, or a personal computer. The wearable device may be a digital electronic device that can be attached to or removed from the user's body or clothing. The wearable device may have various forms, such as, for example, T-shirts, glasses, bracelets, watches, and shoes. By way of example, a wearable device may be a smart watch that the user can wear on the wrist, a Bluetooth earphone that the user can wear on the ear, or smart glasses that the user can wear like glasses. It may include a head mounted display (HMD) such as. Smart glasses, one of the external devices 20, can display a desired image in a processed space based on extended reality (eXtended Reality) technology. The space (e.g., mirroring display area) processed based on the extended reality technology is, for example, a virtual image (e.g., mirroring image) in the display area (or display window) of the external device 20 that can display a real image. ) may be part or the entire display area to display. The extended reality technology may be a technology that supports at least one of virtual reality (VR), augmented reality (AR), or mixed reality (MR). The mirroring screen may expand the screen of the app executed by the source device 10 on the virtual display. As an example, the screen expansion in the mirroring function may include a VR expansion screen, an AR expansion screen, or an MR expansion screen, depending on the applied technology.

The source device 10 and the external device 20 may be connected for a mirroring function based on a protocol (hereinafter referred to as 'communication protocol') that supports a certain communication method. The connection for the mirroring function may, for example, allocate a wireless channel to transmit or receive data according to the mirroring function (e.g., mirroring data 31 or event information 33 in FIG. 2). The communication protocol is used to perform communication in a network environment, for example, a short-range communication network (e.g., the first network 1898 in FIG. 18) or a long-distance communication network (e.g., the second network 1899 in FIG. 18). It may be a prepared protocol. The protocol prepared for the short-range communication network 1898 may include, for example, Bluetooth, WiFi direct, or IrDA. The long-distance communication network 1899 may include, for example, legacy telecommunication networks such as cellular networks, 5G networks, next-generation communications networks, the Internet, or computer networks (e.g., LANs or WANs).

By performing a connection procedure, the source device 10 and the external device 20 transmit data (e.g., mirroring data 31 or event information 33 in FIG. 2) to the other device, or transmit data from the other device. A state may be entered in which a wireless channel capable of receiving data (e.g., mirroring data 31 or event information 33) is formed. The connection procedure may be triggered, for example, by the external device 20 requesting a mirroring function from the source device 10. However, the mirroring function request by the external device 20 may be made, for example, after a wireless channel is established with the source device 10 through a connection procedure.

The source device 10 may transmit or receive mirroring-related data for the mirroring function 30 to the external device 20 through a wireless channel allocated by a connection procedure. For example, the source device 10 may transmit mirroring data for a mirroring function (e.g., mirroring data 31 in FIG. 2) to the external device 20. For example, the mirroring data 31 may not be transmitted as an individual layout, code, or function unit, but rather as image data for the screen itself displayed on the virtual display of the source device 10.

For example, the external device 20 may store event information (e.g., event information 33 in FIG. 2) obtained through interaction with a user (e.g., interaction 25 in FIG. 2). It can be transmitted to the source device 10. The user can control the screen to be displayed in the expanded space (eg, the mirroring display area 23 in FIG. 2) through the interaction 25. The interaction 25 may be performed, for example, by tracking hand movements by the user (hand tracking) or tracking the user's gaze (eye tracking).

The external device 20 uses the mirroring data provided from the source device 10 to display a display area (e.g., the mirroring display area in FIG. 2) prepared for the mirroring function on the display (e.g., the display module 210 in FIG. 3). (23)) A mirrored image can be displayed in whole or in part. The mirroring screen on the external device 20 may be a screen in which a mirroring image is displayed on all or part of the mirroring display area 23 .

According to one example, the external device 20 displays a first display area (e.g., first display area 620 of FIG. 6) and/or at least one extended display area (e.g., extended display area 640 of FIG. 6). )) The mirroring display area 23 can be allocated to include. For example, the external device 20 may allocate the mirroring display area 23 to include a first display area (eg, the first display area 620 in FIG. 6). For example, the external device 20 may allocate the mirroring display area 23 to include at least one extended display area (eg, the extended display area 640 in FIG. 6). For example, the external device 20 includes a first display area (e.g., first display area 620 in FIG. 6) and at least one extended display area (e.g., extended display area 640 in FIG. 6). The mirroring display area 23 can be allocated to include.

According to one example, the external device 20 may provide a buffer (or buffer area) for temporarily storing or rendering a mirrored image. The external device 20 improves the buffer or buffer area to provide a mirroring image in addition to the first display area (e.g., the first display area 620 in FIG. 6) and an extended display area (e.g., the extended display area in FIG. 6). (640)) can be implemented. For example, the external device 20 may display mirroring data (e.g., first to third mirroring data 720a of FIG. 7) in the first display area (e.g., first display area 620 of FIG. 6). , 720b, 720c)) may be provided with a first buffer (or first buffer area) (eg, first buffer 241 in FIG. 3) for temporary storage or rendering. For example, the external device 20 may display extended mirroring data (e.g., the first to third extended mirrors of FIG. 7) on at least one extended display area (e.g., the extended display area 640 of FIG. 6). A second buffer (or second buffer area) (for example, the second buffer 243 in FIG. 3) may be provided to temporarily store or render data (731, 733, 735). For example, the external device 20 analyzes information (e.g., motion information according to the user's gesture) obtained through interaction with the user (e.g., interaction 25 in FIG. 2), and analyzes the analysis result. Extended mirroring data temporarily stored in the second buffer (or second buffer area) (e.g., the second buffer 243 in FIG. 3) (e.g., the first to third extended mirroring data 731, 733 in FIG. 7, 735)) can be used to display an extended mirroring image in at least one extended display area (e.g., the extended display area 640 of FIG. 6).

When a request for termination of the mirroring function occurs, the source device 10 and/or the external device 20 may release all resources allocated for the mirroring function and then terminate the mirroring function. For example, the source device 10 may discard at least one virtual display that was created for a mirroring function. For example, the external device 20 may release the display area (e.g., the mirroring display area 23 in FIG. 2) that was allocated for the mirroring function. For example, the external device 20 may be a buffer (e.g., first buffer 21 or second buffer 23) or buffer area (e.g., first buffer area or second buffer) used for the mirroring function. area) can be deallocated.

FIG. 2 is a diagram illustrating a procedure for a mirroring function in a mirroring system (eg, mirroring system 1 of FIG. 1) according to an embodiment.

Referring to FIG. 2 , the mirroring system 1 may include a source device 10 and at least one external device 20. The source device 10 may provide one or more virtual displays to provide a mirrored image to the external device 20 for a mirroring function. For example, the virtual display may correspond to an area allocated to generate image data for displaying a mirroring screen in memory.

The source device 10 may share its app list with the external device 20. For example, the app list may be a list of apps installed on the source device 10. The source device 10 generates an initial mirroring image including icons of apps to provide a mirroring function on a virtual display (e.g., the first virtual display 11 and the second virtual display 13), and Initial mirroring data 31 corresponding to the mirroring image may be provided to the external device 20. The app list may include, for example, a two-dimensional app (2D app), a three-dimensional widget app (3D widget app), or a three-dimensional immersive app (3D immersive app). The two-dimensional app may be, for example, an app that provides flat images. The 3D widget app may be, for example, a 3D app that can be deployed in large numbers in a processed space (AR space, VR space, extended reality space). The 3D realistic app may be, for example, a 3D app that allows one 3D app to control the entire processed space (AR space, VR space, extended reality space).

The external device 20 may obtain information about apps that will use the mirroring function from the app list shared by the source device 10. The external device 20 may receive initial mirroring data 31 from the source device 10. The external device 20 uses the information about the app and/or the initial mirroring data 31 to create an initial mirroring screen to use the mirroring function on all or part of the mirroring display area 23 provided in the display module 110. It can be displayed.

The source device 10 may initiate a mirroring function in response to an app execution request from the external device 20. When the mirroring function is initiated, the source device 10 displays an encoder and/or a virtual display (e.g., the first virtual display 11 and/or Alternatively, a second virtual display 13) can be created. The source device 10 can create multiple virtual displays to run multiple apps at the same time. For example, the source device 10 may create and operate one to six virtual displays.

The source device 10 can launch a corresponding app on the virtual display. The source device 10 may transmit mirroring data 31 including a mirroring image that can be displayed on a virtual display by executing an app to the external device 20. For example, the mirroring data 31 may be a virtual screen displayed on the virtual display generated by the source device 10 as is in the mirroring display area 23 allocated to the display window 29 of the display of the external device 20. ) can include data for mirroring.

The external device 20 displays or projects a real space image through the entire area of the display window 29 of the display, and allocates a mirroring display area 23 to a partial area of the display window 29 to determine the source device ( The mirroring image 27 can be displayed using the mirroring data 31 provided from 10). For example, the mirroring display area 23 may include a plurality of display areas (eg, the first display area 620 or at least one extended display area 640 in FIG. 6).

When an event is generated through interaction 25 with a user, the external device 20 may transmit event information 33 corresponding to the generated event to the source device 10. For example, if the external device 20 determines that it can process the generated event on its own, it can process it directly. The event may be generated, for example, by an interaction 25 such as the user touching a specific position of the mirroring image displayed in the mirroring display area 27 or scrolling.

When the source device 10 receives event information from the external device 20, it can process an operation corresponding to the event information on the corresponding app running on the virtual display 11 and 13. The source device 10 may provide mirroring data 31 corresponding to images to be displayed on the virtual displays 11 and 13 to the external device 20 according to operation processing corresponding to the event information.

FIG. 3 is a configuration diagram of a mirroring system (eg, the mirroring system of FIG. 1) for supporting screen mirroring, according to an embodiment.

Referring to FIG. 3, a source device (e.g., source device 10 in FIG. 1) and an external device (e.g., external device 20 in FIG. 1) are connected through a network 40 based on a predetermined communication protocol. You can. When the source device 10 and the external device 20 are connected through the network 40, a communication channel 70 according to a predetermined communication protocol may be allocated. The communication channel 70 may be, for example, a wireless channel or a wired channel.

As an example, the source device 10 may include a display module 110, a processor 120, an input/output unit 130, a memory 140, or a communication module 150.

The display module 110 may include a display panel or a touch panel. The display panel can visually provide information to the outside of the source device 10 (eg, a user). The display module 110 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. The touch panel may include a touch sensor configured to detect a touch by a user, or a pressure sensor configured to measure the intensity of force generated by the touch. The operation of the display module 110 can be controlled by electrical connection with the processor 120. The display panel or touch panel included in the display module 110 can have its operation independently controlled by electrical connection with the processor 120. In one embodiment, the display panel may visually provide display information to be displayed to the outside (eg, a user) under the control of the processor 120.

The processor 120 executes software (e.g., program 1840 in FIG. 18) to operate an electrically connected display module 110, input/output unit 130, memory 140, or communication module 150. It can control at least one other component (e.g., hardware or software component) and perform various data processing or operations. The processor 120, as at least part of data processing or computation, stores commands or data received from other components (e.g., display module 110, communication module 150, or input/output unit 127) in memory ( 140) (e.g., volatile memory), or the commands or data stored in the memory 140 may be processed, and the processed result data may be stored in the memory 140.

The input/output unit 130 may receive commands or data to be used in a component of the source device 10 (eg, the processor 120) from outside the source device 10 (eg, a user). The input/output unit 130 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The communication module 150 may support establishment of a direct (eg, wired) communication channel or wireless communication channel with the external device 20, and communication through the established communication channel. The communication module 150 operates independently of the processor 120 and may include one or more communication processors that support direct (eg, wired) communication or wireless communication. The communication module 150 is, for example, a wireless communication module (e.g., the wireless communication module 1892 of FIG. 18) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module. ) or a wired communication module (e.g., the wired communication module 1894 of FIG. 18) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module is a network 40 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct or infrared data association (IrDA), or a legacy cellular network, 5G network, next-generation communication network, Internet, Alternatively, it may communicate with the external device 20, which is an external electronic device, through a computer network (e.g., a long-distance communication network such as LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips).

The memory 140 is used by at least one component (e.g., display module 110, processor 120, communication module 150, or input/output unit 127) of the source device 10. A variety of data can be stored. The data may include, for example, software (e.g., program 1840 of FIG. 18) and input data or output data for commands related thereto. The memory 140 may include, for example, volatile memory or non-volatile memory.

According to one example, the processor 120 may transmit its app list to the external device 20 through the communication module 150, thereby sharing the app list with the external device 20. The app list may be a list of apps installed on the source device 10. For example, when the processor 120 controls the communication module 150 to connect the external device 20 and then receives a mirroring request from the external device 20, the app list is sent to the communication module. It can be transmitted to the external device 20 through (150). For example, the processor 120 generates an initial mirroring image including icons of apps that will provide a mirroring function on the virtual display (e.g., the first virtual display 11 and the second virtual display 13 in FIG. 2). And, initial mirroring data 31 corresponding to the initial mirroring image can be provided to the external device 20.

According to one example, the processor 120 may initiate a mirroring function in response to an app execution request transmitted from the external device 20 through the communication module 150. When the mirroring function is initiated, the processor 120 generates an encoder and/or a virtual display (e.g., the first virtual display 11 of FIG. 2) for each one or multiple apps requested to be executed by the external device 20. and/or the second virtual display 13) may be generated. The processor 120 can create multiple virtual displays to execute multiple apps at the same time. The processor 120 may create and operate, for example, one to six virtual displays. The processor 120 can launch a corresponding app on the virtual display. The processor 120 may transmit mirroring data 31 including a mirroring image that can be displayed on a virtual display by executing an app to the external device 20. For example, the mirroring data 31 may be a mirroring display area 23 allocated to the display window 29 of the display of the external device 20 as is the virtual screen displayed on the virtual display generated by the processor 120. ) can include data for mirroring.

According to one example, when the processor 120 receives event information from the external device 20 through the communication module 150, the processor 120 performs an operation corresponding to the event information on the corresponding app running on the virtual displays 11 and 13. can be processed. The source device 10 may provide mirroring data 31 corresponding to images to be displayed on the virtual displays 11 and 13 to the external device 20 according to operation processing corresponding to the event information.

As an example, the external device 20 may include a display module 210, a processor 220, a sensor module 230, a memory 240, or a communication module 250. The memory 240 may include a first buffer 241 or a second buffer 243. Although not shown, at least one of the first buffer area or the second buffer area may be allocated to a register within the processor 220. The first buffer 241 or the first buffer area may temporarily store mirroring data provided from the source device 10. Mirroring data temporarily stored in the first buffer 241 or the first buffer area may be discarded after another mirroring image is displayed in the first display area.

The display module 210 may visually provide information (e.g., real image information or mirroring image information) to the outside of the external device 20 (e.g., a user). The display module 210 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. The operation of the display module 210 can be controlled by electrical connection with the processor 220.

The processor 220 executes software (e.g., program 1840 in FIG. 18) to run at least one electrically connected display module 210, sensor module 230, memory 240, or communication module 250. It can control other components (e.g. hardware or software components) and perform various data processing or operations. The processor 220, as at least part of data processing or computation, processes commands or data received from other components (e.g., display module 210, sensor module 230, or communication module 250) into memory 240. (eg, volatile memory), or the commands or data stored in the memory 240 can be processed, and the processed result data can be stored in the memory 240.

The sensor module 230 detects the operating state (e.g., power or temperature) of the external device 20, the external environmental state (e.g., user state), or the movement of the hand or gaze for interaction with the user, An electrical signal or data value corresponding to the sensed state can be generated. The sensor module 230 includes, for example, a gesture sensor, gyro sensor, barometric pressure sensor, magnetic sensor, acceleration sensor, grip sensor, proximity sensor, color sensor, IR (infrared) sensor, biometric sensor, temperature sensor, and humidity sensor. , or may include an illumination sensor.

The communication module 250 may support establishment of a direct (eg, wired) communication channel or wireless communication channel with the source device 10, and communication through the established communication channel. The communication module 250 operates independently of the processor 220 and may include one or more communication processors that support direct (eg, wired) communication or wireless communication. The communication module 250 is, for example, a wireless communication module (e.g., the wireless communication module 1892 of FIG. 18) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module. ) or a wired communication module (e.g., the wired communication module 1894 of FIG. 18) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module is a network 40 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct or infrared data association (IrDA), or a legacy cellular network, 5G network, next-generation communication network, Internet, Alternatively, it may communicate with the source device 10, which is an external electronic device, through a computer network (e.g., a long-distance communication network such as LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips).

The memory 240 is used by at least one component (e.g., display module 210, processor 220, communication module 250, or input/output unit 127) of the external device 20. A variety of data can be stored. The data may include, for example, software (e.g., program 1840 of FIG. 18) and input data or output data for commands related thereto. The memory 240 may include, for example, volatile memory or non-volatile memory.

The external device 20 may further include a camera module. The camera module can capture still images and moving images. The camera module may include one or more lenses, image sensors, image signal processors, or flashes. As an example, the camera module can be used for interaction with the user.

According to one example, the processor 220 may include face tracking, hand tracking, or motion tracking functions. The processor 220 may perform interaction with the user using at least one of face tracking, hand tracking, or motion tracking functions. The face tracking function extracts data of the part corresponding to the face from the user image captured by the camera module and can track changes in facial expressions based on the extracted data. The hand tracking function extracts data of the part corresponding to the hand from the user image captured by the camera module and tracks hand movement based on the extracted data. The hand tracking function may track hand movement based on sensing information provided from the sensor module 230 or sensing information provided from the source device 10. The motion tracking function can track the user's movement from the user image captured by the camera module. The motion tracking function may track the user's movement based on sensing information provided from the sensor module 230 or sensing information provided from the source device 10.

According to one example, the processor 220 may control the communication module 250 to connect the source device 10 and then request a mirroring function. The processor 220 may receive an app list from the source device 10 through the communication module 250 in response to the mirroring request. The processor 220 may obtain information about apps that will use the mirroring function from the app list. The processor 220 may receive initial mirroring data 31 from the source device 10. The processor 220 may use the information about the app and/or the initial mirroring data 31 to display an initial mirroring screen using the mirroring function on all or part of the mirroring display area provided in the display module 210. there is.

According to one example, the processor 220 displays or projects a real space image through the entire area of the display window of the display module 210, and displays the source device 10 in a mirroring display area allocated to a partial area of the display window. A mirroring image 27 using mirroring data 31 provided from can be displayed. For example, the mirroring display area may include a plurality of display areas (eg, the first display area 620 or at least one extended display area 640 in FIG. 6). The processor 220 may display a mirroring image on the first display area 620 using mirroring data transmitted in real time from the source device 10. The mirroring data transmitted in real time may be temporarily stored in the first buffer 241 or the first buffer area allocated to the memory 240. Mirroring data temporarily stored in the first buffer 241 or the first buffer area may be updated with the next received mirroring data when the mirroring image exposed in the first display area 620 is changed. The processor 220 may display an extended mirroring image in the extended display area 640 using mirroring data that is displayed in the first display area 620 or has a history of being displayed. Mirroring data that is displayed in the first display area 620 or has a history of being displayed may be temporarily stored in the second buffer 243 or a second buffer area allocated to the memory 240. The mirroring data stored in the second buffer 243 or the second buffer area may be discarded in oldest order or by user selection, taking into account free space in the storage area.

According to one example, when an event is generated through interaction 25 with a user, the processor 220 may transmit event information 33 corresponding to the generated event to the source device 10. For example, if the processor 220 determines that the generated event can be processed on its own, the processor 220 may process it directly. The event may be generated, for example, by an interaction 25 such as the user touching a specific position of the mirroring image displayed in the mirroring display area 27 or scrolling.

FIG. 4 is a diagram illustrating the functional configuration of a source device (eg, source device 10 of FIG. 1) that supports screen mirroring, according to an embodiment.

Referring to FIG. 4, the source device 10 may include a transfer management module (Transfer Manager) 410, a container management module (App Execution Container Manager) 420, or a system service module (System Services) 430. there is.

The transmission management module 410 may be responsible for receiving an event request, mirroring request, or response (ACK) message from an external device (eg, the external device 20 of FIG. 1). The transmission management module 410 may transmit mirroring data for the mirroring function (e.g., mirroring data 310 in FIG. 2) to the external device 20.

The container management module 420 may play a role in controlling mirroring logic. For example, when an app execution request is provided from the external device 20 through the transmission management module 410, the container management module 420 creates an App Execution Container (App Execution Container) for executing the requested app. 420-1, 420-2, 420-N) can be produced. The container management module 420 may own or manage one or more container units 420-1, 420-2, and 420-N for the mirroring function. When an app-related request (e.g., event request 33 in FIG. 2) is provided from the external device 20 through the transmission management module 410, the container management module 420 stores the app for which the request was received. Requests and/or data can be distributed to the container using the identifier (container id) of the container created in response. For example, when an event is generated in the system service module 430, the container management module 420 may specify a target container using a display identifier (display Id) or a task identifier (task Id). The container management module 420 may send an event to a specified target container or transmit the event to the external device 20 through the transmission management module 410.

The container units 420-1, 420-2, and 420-N may control each mirroring session for the mirroring function. The container units 420-1, 420-2, and 420-N may have a unique identifier (container ID). The container units 420-1, 420-2, and 420-N may form a mapping relationship with a remote app container manager provided in the external device 20 using a unique identifier. The container units 420-1, 420-2, and 420-N may have a package name, task identifier (task id), user identification information (User ID), etc. of the app being executed. Each of the container units (420-1, 420-2, 420-N) has a virtual display (421-1, 421-2, 421-N) and an input surface (423-1) inside. , 423-2, 423-N) or a video encoder (425-1, 425-2, 425-N).

As an example, the container units 420-1, 420-2, and 420-N generate video encoders 425-1, 425-2, and 425-N, and the video encoders 425-1 and 425-2 , A virtual display can be created by designating the input surfaces 423-1, 423-2, and 423-N of 425-N as the surfaces of virtual displays 421-1, 421-2, and 421-N. . The container units 420-1, 420-2, and 420-N can execute apps corresponding to the virtual displays 421-1, 421-2, and 421-N.

According to one example, when an app is executed on the virtual displays 421-1, 421-2, and 421-N, the surface finger unit 431 included in the system service module 430 moves the app screen to the corresponding virtual display. It can be displayed on the input surface of . The image or screen mirroring data displayed on the virtual displays 421-1, 421-2, and 421-N by the surface finger unit 431 is sent to the video encoders 425-1, 425-2, and 425-N. It can be delivered and compressed. The compressed mirroring data may be transmitted to the external device 20 through the transmission management module 410. The external device 20 can output a mirroring screen using the transmitted mirroring data.

The system service module 430 may include a Remote App Mode Service (435), a Surface Flinger (431), or a Display Manager (433). The remote app mode service unit 435 can perform functions that require system permission, such as executing apps on the virtual displays 421-1, 421-2, and 421-N or terminating apps.

According to the above, the source device 10 executes each of a plurality of apps on an independently created virtual display, and transmits a screen corresponding to the execution of the app on each of the virtual displays to the external device 20, allowing multiple apps to be displayed at the same time. It can be made to float in virtual space. The execution state of each app executed on the virtual display may be managed by the container management module 420. For example, the container management module 420 can manage each app in units of containers including a class responsible for input processing and a video encoder.

FIG. 5 is a diagram illustrating the functional configuration of an external device (eg, the external device 20 of FIG. 1) that supports screen mirroring, according to an embodiment.

Referring to FIG. 5, the external device 20 is a transfer management module (Transfer Manager) 510, a remote app container management module (Remote App Container Manager) 520, or a container window management module (Container Window Manager) 530. may include.

The remote app container management module 520 may create, dismantle/destroy, or manage one or more remote app container units 520-1, 520-2, and 520-N. The remote app container management module 520 transmits the mirroring data transmitted from the source device 10 through the transmission management module 510 to the corresponding remote app container unit (Remote App Container 520-1, 520-2, 520). -N) can perform the role of distribution. The remote app container management module 520 may be provided to correspond to the container management module 420 of the source device 10.

The remote app container units 520-1, 520-2, and 520-N may be responsible for one screen mirroring session. The remote app container units 520-1, 520-2, and 520-N may have a unique identifier (Container id). The remote app container units (520-1, 520-2, 520-N) are among the container units (420-1, 420-2, 420-N) of the source device 10 using an identifier (Container id). It may correspond to one specific container unit.

The remote app container units 520-1, 520-2, and 520-N display a mirrored image in the first display area or extended area using the resolution and/or DPI determined by the container window management module 530. can do.

The remote app container unit (520-1, 520-2, 520-N) includes a video decoder (521-1, 521-2, 521-N), an output surface (Out Surface) (523-1, 523-2, 523-N) or Surface View (525-2, 525-2, 525-N). The remote app container unit (520-1, 520-2, 520-N) responds to the execution of the app for the mirroring function in the source device 10 by sending a video decoder (521-1, 521-2, 521-N) ) or surface views (525-2, 525-2, 525-N) can be created. The video decoders 521-1, 521-2, and 521-N may perform decoding on compressed mirroring data transmitted from the source device 10 through the transmission management module 510. The output surfaces 523-1, 523-2, and 523-N display the mirroring data decompressed by the video decoders 521-1, 521-2, and 521-N into surface views 525-2 and 525. -2, 525-N). The surface views 525-2, 525-2, and 525-N display a mirrored image in the first display area using mirroring data transmitted through the output surfaces 523-1, 523-2, and 523-N. Alternatively, you can display an extended mirroring image in the extended display area.

The container window management module 530 determines the resolution and/or DPI for the mirroring screen corresponding to the app for which the mirroring function is to be provided, and sends this to the remote app container units 520-1, 520-2, and 520-N. It can perform the role provided by .

The transmission management module 510 transmits a mirroring request or response (ACK) message to the source electronic device 10, or receives mirroring-related data from the source electronic device 10 and receives the remote app container management module ( 520).

FIG. 6 is an example of a screen display of a virtual space on an external device (eg, the external device 20 of FIG. 1) according to an embodiment.

Referring to FIG. 6, the external device 20 may include a display 600 that displays a virtual space. The display 600 may be, for example, a transparent display. The transparent display may have a transparent window. For example, if the external device 20 is smart glasses, which are one of the wearable devices, the user can view the actual image 610 of the real space through the transparent window. The actual image 610 may be, for example, a perspective image or an image captured by a camera. The image captured by the camera and displayed on the display 600 may be processed or modified as needed.

According to one example, the display 600 may include a display area (eg, the mirroring display area 23 in FIG. 2) for providing a mirroring screen according to a mirroring function. The mirroring display area 23 may include a first display area 620 or an extended display area 640. The extended display area 640 may include one or a plurality of extended display areas 641 and 642 in response to an event 630 generated by user interaction, such as a user's gesture.

In the first display area 620, a mirroring image 650 is displayed based on mirroring data (e.g., mirroring data 310 in FIG. 2) last provided from a source device (e.g., source device 10 in FIG. 1). It can be. The last provided mirroring data 31 may be temporarily stored in a first buffer (eg, first buffer 241 in FIG. 3) or a first buffer area. When the mirroring image 650 is displayed in the first display area 620, the first buffer 241 or the mirroring data 31 temporarily stored in the first buffer area is stored in the second buffer 243 or the second buffer. It can also be stored in the area. The previous mirroring data stored in the second buffer 243 or the second buffer area is an extended mirroring image (extended mirroring image) displayed in the extended display area 640 in response to an event 630 generated by interaction with the user, such as a user's gesture. 660). The extended mirroring image 660 may be, for example, a first extended mirroring image 661 displayed on the first extended display area 641 or a second extended mirrored image 662 displayed on the second extended display area 642. may include.

As described above, in response to the user's request, an extended mirroring image 660 is displayed in the extended display area 640 of the display 600 using mirroring data previously delivered from the source device 10 and stored temporarily. can do. This not only does not consume additional resources to expand the mirroring screen, but also reduces power consumption by providing an efficient rendering procedure.

Although the drawing shows that the extended display area 640 is formed in the upper direction of the first display area 620, it may not be limited thereto. For example, the direction in which the expanded display area 640 is provided can be determined by considering event information obtained through interaction with the user. The direction in which the extended display area 640 is provided may be, for example, one of the left, right, or downward directions with respect to the first display area 620 . For example, the expanded display area 640 may be expanded in the rear direction based on the first display area 620. For example, the expanded display area 640 may be expanded in a direction that is easy to expand (eg, a direction that is easy to expand in the display 600). The direction in which the expanded display area 640 expands is not limited to one direction, but may also expand radially in multiple directions. The number of expanded display areas 640 that can be expanded in response to event generation in the display 600 may be limited by the number of mirroring data to be used to obtain an expanded mirroring image. The second buffer 243 or the second buffer area that can limit the number of mirroring data can be increased or decreased as needed. The expanded display area 640 allows the function and scope of the app for which the mirroring function is provided to be further expanded. Since the expanded display area 640 can utilize the entire area within the field of view (FOV) in the field of view of the user wearing the external device 20, the mirroring screen can be displayed by enlarging or expanding it.

As described above, a more realistic experience can be provided to the user by expanding the size of the mirroring screen on which the mirroring image is displayed each time an event is generated through interaction with the user.

In FIG. 6 , one first display area 620 is assumed. However, as a plurality of first display areas are exposed, a situation may occur where the display area for providing an expanded display area is insufficient. As an example, the external display 20 changes the layout of arranging the first display area and/or extended display area displayed in the display area in response to the user's request input through interaction with the user, or changes some of the display areas. It can be operated to remove or hide.

According to one example, in FIG. 6, it is assumed that the real image 610 is actually visible to the user through the transparent window of the display 600, but the external device 20 uses a camera to consider the user's gaze. An image of the front may be captured and the captured image may be exposed through the display area.

According to one example, in FIG. 6, it is assumed that the real image 610 is actually visible to the user through the transparent window of the display 600, but the external device 20 exposes an arbitrarily prepared image through the display area. You may.

According to one example, the external device 20 may divide the display area 600 into an interest area and a non-interest area. For example, the area of interest can be designated in the display area 600 by considering the area of the user's gaze or the surroundings the user looks at through eye tracking. In this case, the external device 20 may expand the extended display area 640 in a space divided from the display area to the area of interest. Even if it is not the extended display area 640, performance can be improved by operating a static image within the area of interest through a second buffer.

FIG. 7 is an example diagram of expanding a mirroring screen in an external device (e.g., external device 20 of FIG. 1) of a mirroring system (e.g., mirroring system 1 of FIG. 1) according to an embodiment.

Referring to FIG. 7, when the external device 20 receives first mirroring data 720a from a source device (e.g., source device 10 of FIG. 1), the external device 20 transmits the received first mirroring data 720a. It may be temporarily stored in the first buffer (e.g., first buffer 241 in FIG. 3) or the first buffer area. The external device 20 may allocate a first display area to the display and display a first mirroring image 710a in the allocated first display area using the first mirroring data 720a. When the external device 20 displays the first mirroring image 710a in the first display area, the first buffer 241 or the first mirroring data 720a temporarily stored in the first buffer area is stored in the second buffer. (For example, the second buffer 243 in FIG. 3) or may be stored as first extended mirroring data 731 in the second buffer area.

When the external device 20 receives the second mirroring data 720b from the source device 10, the received second mirroring data 720b is stored in the first buffer (720b) instead of the first mirroring data 720a. Example: It can be temporarily stored in the first buffer 241 of FIG. 3) or in the first buffer area. The external device 20 may display a second mirroring image 710b using the second mirroring data 720b in the first display area allocated to the display. When displaying an extended mirroring image is requested 750b through interaction with a user, the external device 20 allocates an extended display area to the display and stores a second buffer (e.g., the second buffer in FIG. 3) in the extended display area. The first extended mirroring image 740b can be displayed using the first extended mirroring data 731 stored in the 2 buffer 243) or the second buffer area. When the external device 20 displays the second mirroring image 710b on the first display area, the second mirroring data 720b temporarily stored in the first buffer 241 or the first buffer area is stored in the second buffer. (For example, the second buffer 243 in FIG. 3) or may be stored as second extended mirroring data 733 in the second buffer area.

When the external device 20 receives the third mirroring data 720c from the source device 10, the received third mirroring data 720c is stored in the first buffer (720c) instead of the second mirroring data 720b. Example: It can be temporarily stored in the first buffer 241 of FIG. 3) or in the first buffer area. The external device 20 may display a third mirroring image 710c using the third mirroring data 720c in the first display area allocated to the display. When the display of an extended mirroring image is requested through interaction with the user (750c, 760c), the external device 20 allocates an additional extended display area to the display and displays a second display area in the existing and additionally allocated extended display areas. A first extended mirroring image and a second extended mirroring image using the first extended mirroring data 731 and the second extended mirroring data 733 stored in a buffer (e.g., the second buffer 243 in FIG. 3) or the second buffer area. A mirroring image 740c can be displayed. When the external device 20 displays the third mirroring image 710c on the first display area, the third mirroring data 720c temporarily stored in the first buffer 241 or the first buffer area is stored in the second buffer. (For example, the second buffer 243 in FIG. 3) or it can be stored as third extended mirroring data 735 in the second buffer area.

FIG. 8 is a diagram illustrating a processing procedure for providing a mirroring function in a mirroring system (eg, mirroring system 1 of FIG. 1) according to an embodiment.

Referring to FIG. 8, the mirroring system 1 may include at least two electronic devices. At least two electronic devices included in the mirroring system 1 include, for example, a first electronic device (e.g., a source device (e.g., the source device (e.g., FIG. 10)) and at least one second electronic device (eg, the external device 20 of FIG. 1) that displays the mirroring data 31. The first electronic device 10 may be an external electronic device to the second electronic device 20. The second electronic device 20 may be an external electronic device to the first electronic device 10. The first electronic device 10 may be a digital device, such as a smartphone or tablet PC, in which programs such as apps can be installed or executed. The second electronic device 20 may be, for example, an external device 20 such as smart glasses, as well as a digital device such as a smartphone with its own display, a tablet PC, or a personal computer. For convenience of description below, the first electronic device 10 is referred to as a source device 10, and the second electronic device 20 is referred to as an external device 20. However, operations according to embodiments to be described later can be equally applied to electronic devices including a display, such as the source device 10 or the external device 20.

The source device 10 and the external device 20 may perform a connection procedure in operation 811. The connection procedure may include an operation in which at least one of the source device 10 or the external device 20 confirms the other device and then requests a connection. The connection procedure may include an operation in which at least one of the source device 10 or the external device 20 accepts the connection in response to a connection request from the other device. Detailed operations according to the connection procedure may be defined by protocols (hereinafter referred to as 'communication protocols') provided for each communication method supported by the source device 10 or the external device 20. The communication protocol is used to perform communication in a network environment, for example, a short-range communication network (e.g., the first network 1898 in FIG. 18) or a long-distance communication network (e.g., the second network 1899 in FIG. 18). It may be a prepared protocol. The protocol prepared for the short-range communication network 1898 may include, for example, Bluetooth, WiFi direct, or IrDA. The long-distance communication network 1899 may include, for example, legacy telecommunication networks such as cellular networks, 5G networks, next-generation communications networks, the Internet, or computer networks (e.g., LANs or WANs). By performing a connection procedure, the source device 10 and the external device 20 transmit data (e.g., mirroring data 31 or event information 33 in FIG. 2) to the other device, or transmit data from the other device. A state may be entered in which a wireless channel capable of receiving data (e.g., mirroring data 31 or event information 33) is formed. The connection procedure may be triggered, for example, by the external device 20 requesting a mirroring function from the source device 10. However, the mirroring function request by the external device 20 may be made, for example, after a wireless channel is established with the source device 10 through a connection procedure.

When a wireless channel is established by performing the connection procedure, the source device 10 may monitor whether an app execution event is generated in operation 813. For example, when execution of a specific app is requested from the external device 20, the source device 10 may determine that an app execution event has been generated. To this end, the source device 10 may provide app information (e.g., app list) that it can execute to the external device 20 in response to a mirroring function request from the external device 20. For example, the app information may be included in mirroring data generated using a screen (e.g., a virtual screen) on which icons created when an app is installed in the source device 10 are displayed.

When the external device 20 receives the app information, it can configure and display a mirroring screen (e.g., the mirroring display area 23 in FIG. 6) using the app information. The external device 20 may recognize that a user input requesting execution of a specific app is generated on the mirroring screen displayed in the mirroring display area 23 through interaction with the user.

When the external device 20 requests provision of a mirroring function for a specific app, it can share it with the source device 10. Accordingly, the source device 10 and the external device 20 can perform mirroring and event processing operations in operation 817. The mirroring and event processing operations are, for example, performed by the external device 20 to expose a mirrored image on the first display area 620 using mirroring data provided by the source device 10. Can include actions. For example, the mirroring and event processing operations may be performed by recycling mirroring data that the external device 20 is displaying in the first display area 620 or has previously been displayed as a mirroring image, thereby creating at least one extension. It may include an operation performed to expose an extended mirroring image in the display area 640.

In operation 819, the external device 20 may determine whether an app termination event is generated. The app termination event may be generated, for example, when a user's input through interaction with the external device 20 indicates a request to terminate the mirroring function.

When the app termination event is generated, the external device 20 may transmit an app termination request to the source device 10 in operation 821. When requesting termination of the app, the external device 20 may also transmit identification information indicating the app to be terminated.

When the source device 10 receives an app termination request from the external device 20, the source device 10 may terminate the corresponding app and/or related task in operation 823. In operation 825, the source device 10 may release related resources for the mirroring function of the app whose termination has been requested. As an example, the source device 10 may release a container unit (e.g., container units 420-1, 420-2, and 420-N in FIG. 4) prepared in response to an app for which release has been requested. When the container units 420-1, 420-2, and 420-N are released, a virtual display constituting the container units 420-1, 420-2, and 420-N (e.g., the virtual display in FIG. 4 ( 421-1, 421-2, 421-N), an input surface (e.g., input surface (423-1, 423-2, 423-N) in FIG. 4), and/or a video encoder (e.g., video in FIG. 4) Encoders (425-1, 425-2, 425-N) may also be discarded together.

When the external device 20 transmits an app termination request to the source device 10, in operation 827, the external device 20 may release related resources for the mirroring function of the app for which termination has been requested. As an example, the external device 20 may release a remote app container unit (e.g., remote app container units 520-1, 520-2, 520-N in FIG. 5) prepared in response to an app requested to be released. You can. When the remote app container unit (520-1, 520-2, 520-N) is released, the video decoder (e.g., FIG. 5) constituting the remote app container unit (520-1, 520-2, 520-N) video decoders 521-1, 521-2, 521-N), output surfaces (e.g., output surfaces 523-1, 523-2, 523-N) of FIG. 5, or surface views (e.g., Surface views 525-1, 525-2, 525-N) may also be discarded.

FIG. 9 is a diagram illustrating processing procedures for mirroring and event processing operations in a mirroring system (eg, mirroring system 1 of FIG. 1) according to an embodiment.

Referring to FIG. 9, when the mirroring and event processing operations are enabled, the source device 10 allocates related resources to provide a mirroring function for each of one or a plurality of apps for which the mirroring function has been requested in operation 911. You can. For example, the source device 10 may create a container unit (eg, container units 420-1, 420-2, and 420-N in FIG. 4) for each app. The container units 420-1, 420-2, and 420-N include a virtual display (e.g., the virtual displays 421-1, 421-2, and 421-N of FIG. 4) and an input surface (e.g., the virtual display of FIG. 4). It may include an input surface (423-1, 423-2, 423-N) and/or a video encoder (e.g., the video encoder (425-1, 425-2, 425-N) of FIG. 4). As a result, in operation 911, the source device 10 configures video encoders 425-1, 425-2, and 425-N and virtual displays 421-1, 421-2, and 421-N for each app to perform the mirroring function. ) can be created.

In operation 913, the source device 10 may execute the corresponding app on the virtual displays 421-1, 421-2, and 421-N created for each app. Executing the app may correspond to executing the app in an allocated memory area. The source device 10 executes the app on the virtual displays 421-1, 421-2, and 421-N, thereby displaying the app on the virtual displays 421-1, 421-2, and 421-N. The image can be acquired as a mirrored image. The source device 10 can encode a mirroring image using video encoders 425-1, 425-2, and 425-N created for each app.

The source device 10 may transmit mirroring data including a mirroring image compressed by encoding to the external device 20 in operation 915. The mirroring data may include mirroring metadata to be used to mirror the mirrored image. The mirroring metadata may include, for example, information such as size, resolution, DPI, or display position of the mirrored image. The source device 10 may be, for example, a digital device such as a smartphone or tablet PC on which a program such as an app can be installed or executed.

The external device 20 may allocate related resources to provide the mirroring function for each app to be provided with the mirroring function. For example, the external device 20 may create a remote app container unit (eg, remote app container units 520-1, 520-2, and 520-N in FIG. 5) for each app. The remote app container unit (520-1, 520-2, 520-N) includes a video decoder (521-1, 521-2, 521-N) and an output surface (523-1, 523-2, 523-N). Alternatively, surface views 525-1, 525-2, and 525-N can be created.

In operation 917, the external device 20 may perform decoding on the compressed mirroring data transmitted from the source device 10 in the corresponding video decoders 521-1, 521-2, and 521-N. . In operation 917, the external device 20 may configure a first display area (eg, first display area 620 in FIG. 6) that displays a mirroring image using decompressed mirroring data. The external device 20 may temporarily store the decompressed mirroring data in a first buffer (eg, the first buffer 241 in FIG. 3) in operation 917. As new mirroring data is temporarily stored in the first buffer 241, the external device 20 may discard the mirroring data that was previously temporarily stored. In operation 917, the external device 20 caches the mirroring data used to display the mirrored image in the first display area 620 in a second buffer (e.g., the second buffer 243 in FIG. 3). can do.

The external device 20 may detect an event generated through interaction with a user in operation 919. For example, the interaction may correspond to a user's touch or drag action in the first display area 620.

When the creation of an event through interaction with the user is detected, the external device 20 may determine whether the event is a request for expansion of the mirroring screen in operation 921. If the generated event does not correspond to an expansion request, the external device 20 may transmit information about the input event to the source device 10 in operation 923. If the generated event corresponds to an extension request, the external device 20 acquires an extension mirroring image using mirroring data cached in the second buffer 243 in operation 927, and An extended display area (e.g., extended display area 640 in FIG. 6) can be configured to display a mirroring image.

Upon receiving information about an input event from the external device 20, the source device 10 may process an operation according to the event using the information about the input event in operation 925. The source device 10 may configure mirroring data reflecting the results of processing operations for the event and transmit it to the external device 20.

FIGS. 10A and 10B are control flowcharts for an external device (e.g., external device 20 of FIG. 1) to display a mirrored image in a mirroring system (e.g., mirroring system 1 of FIG. 1) according to an embodiment. am.

Referring to FIGS. 10A and 10B, in operation 1011, the external device 20 requests the source device (e.g., the source device 10 in FIG. 1) to execute the app selected by the user using app information. You can enable the app to run. For example, the app information may be information about apps that can be installed or executed on a source device (eg, source device 10 in FIG. 1). For example, the external device 20 may receive app information from the source device 10. After the source device 10 creates a virtual display and an app is executed on the created virtual display, the external device 20 sends mirroring data of the app screen executed on the virtual display to the source device 10. It can be provided from .

In operation 1013, the external device 20 may determine whether a mirroring event is generated through interaction with the user. When the mirroring event is generated, the external device 20 displays a first display area (e.g., the first display area 620 of FIG. 6) using the mirroring data provided from the source device 10 in operation 1015. ) can display a mirrored image.

When a mirrored image is displayed on the first display area 620, the external device 20 performs extended mirroring on the extended display area allocated to the display (e.g., the extended display area 640 in FIG. 6) in operation 1017. It is possible to determine whether an image is being displayed.

If an extended mirroring image is displayed in the extended display area 640, the external device 20 may reconfigure the extended display area 640 in operation 1019. Reconfiguring the extended display area 640 may include, for example, newly configuring and displaying an extended mirroring image to be displayed on the extended display area 640.

If the extended mirroring image is not displayed in the extended display area 640 or the reconfiguration of the extended display area is completed, the external device 20 displays the mirrored image displayed in the first display area 620 in operation 1021. Caching can be done in two buffers (e.g., the second buffer 243 in FIG. 3).

In operation 1023, the external device 20 may determine whether a user's input is detected through interaction with the user. The user's input may correspond to, for example, touching or dragging a specific location of the first display area 620 or the extended display area 640.

When the user's input is detected, the external device 20 may analyze the user input in operation 1025. For example, the analysis of the user input may be an operation to obtain the type of the user input.

When the analysis of the user input is completed, the external device 20 may determine whether the user's input is a request to display an extended mirroring image in the extended display area 640 in operation 1027. If the user's input does not request display of an extended mirroring image, the external device 20 may determine whether an app operation-related event has been generated in the first display area 620 in operation 1031.

If the user's input requests display of an extended mirroring image, the external device 20 generates an extended mirroring image using mirroring data cached in the second buffer 243 in operation 1029, and generates the extended mirroring image. One extended mirroring image can be displayed in the extended display area 640.

If the user's input is generated in the first display area 620 as an app-related event, the external device 20 may transmit input information according to the user's input to the source device 10 in operation 1033. there is. The input information may be used, for example, by the source device 10 to run the corresponding app.

If the user's input is generated in the extended display area 640 as an app-related event, the external device 20 sends the input information mapped to the extended display area 640 to the source device 10 in operation 1035. It can be sent to . The input information may be used, for example, by the source device 10 to run the corresponding app.

In operation 1037, the external device 20 receives mirroring data created when the source device 10 runs the corresponding app using the input information, and displays the first display area using the received mirroring data. The mirroring image to be displayed at 620 can be updated. For example, the mirroring data may be temporarily stored in the first buffer 241 at the time of reception. The mirroring data may also be temporarily stored in the second buffer 243, for example, after the mirroring image to be displayed on the first display area 620 is updated.

In operation 1039, the external device 20 may determine whether a termination event requesting termination of the mirroring function is generated through interaction with the user. When the termination event is generated, the external device 20 may transmit an app termination request to the source device 10 in operation 1041. For example, the app termination request may cause the source device 10 to abolish the virtual display created to run the app.

FIG. 11A is an example diagram of expanding a mirroring screen on an external device (eg, the external device 20 of FIG. 1) that supports a mirroring function, according to an embodiment.

Referring to FIG. 11A, when the external device 20 displays a mirroring screen including a mirroring image on the first display area 1110, the external device 20 displays a display (e.g., display 600 of FIG. 6). ), the display area (e.g., the extended display area 640 in FIG. 6) for displaying the extended mirroring screen by the extended mirroring image can be expanded around the first display area 1110.

According to one example, the external device 20 may additionally allocate one or more extended display areas in the -x-axis direction 1131 in the first display area 1110, for example. The external device 20 displays an extended mirroring image in the one or more extended display areas using a mirroring image that was previously displayed in the first display area 1110 or is currently displayed in the first display area 1110. can be displayed.

According to one example, the external device 20 may additionally allocate one or more extended display areas in the +x-axis direction 1133 in the first display area 1110 . The external device 20 displays an extended mirroring image in the one or more extended display areas using a mirroring image that was previously displayed in the first display area 1110 or is currently displayed in the first display area 1110. can be displayed.

According to one example, the external device 20 may additionally allocate one or more extended display areas in the +y-axis direction 1121 in the first display area 1110, for example. The external device 20 displays an extended mirroring image in the one or more extended display areas using a mirroring image that was previously displayed in the first display area 1110 or is currently displayed in the first display area 1110. can be displayed.

According to one example, the external device 20 may additionally allocate one or more extended display areas in the -y-axis direction 1123 in the first display area 1110, for example. The external device 20 displays an extended mirroring image in the one or more extended display areas using a mirroring image that was previously displayed in the first display area 1110 or is currently displayed in the first display area 1110. can be displayed.

According to one example, the external device 20 may additionally allocate one or more extended display areas in the +z-axis direction 1141 in the first display area 1110, for example. The external device 20 displays an extended mirroring image in the one or more extended display areas using a mirroring image that was previously displayed in the first display area 1110 or is currently displayed in the first display area 1110. can be displayed.

FIG. 11B is an example diagram of expanding a mirroring screen on an external device (eg, external device 20 of FIG. 1) that supports a mirroring function, according to an embodiment.

Referring to FIG. 11B, the external device 20 does not provide a mirroring screen on the display (e.g., the display 600 of FIG. 6), and the user accesses the source device (e.g., the source device 10 of FIG. 1) through a transparent window. )), the screen 1150 of the source device 10 (hereinafter referred to as the 'source device screen 1150') is visible through the transparent window when checking the display (e.g., the display module 110 in FIG. 3). can be used as the first display area (e.g., the first display area 1110 in FIG. 11A). For example, the external device 20 has a display area (e.g., the extended display area 640 of FIG. 6) to display an extended mirroring screen by a mirroring image centered on the source device screen 1150 visible through a transparent window. ) can be expanded.

According to one example, the external device 20 may additionally allocate one or a plurality of extended display areas in the -x-axis direction 1171 on the source device screen 1150, for example. The external device 20 may display an extended mirroring image on the one or more extended display areas using mirroring data provided by the source device 10.

According to one example, the external device 20 may additionally allocate one or more extended display areas in the +x-axis direction 1173 on the source device screen 1150, for example. The external device 20 may display an extended mirroring image on the one or more extended display areas using mirroring data provided by the source device 10.

According to one example, the external device 20 may additionally allocate one or a plurality of extended display areas in the +y-axis direction 1161 on the source device screen 1150, for example. The external device 20 may display an extended mirroring image on the one or more extended display areas using mirroring data provided by the source device 10.

According to one example, the external device 20 may additionally allocate one or more extended display areas in the -y-axis direction 1163 on the source device screen 1150, for example. The external device 20 may display an extended mirroring image on the one or more extended display areas using mirroring data provided by the source device 10.

According to one example, the external device 20 may additionally allocate one or more extended display areas in the +z-axis direction 1181 on the source device screen 1150, for example. The external device 20 may display an extended mirroring image on the one or more extended display areas using mirroring data provided by the source device 10.

In FIGS. 11A and 11B described above, the location or direction in which the expanded display area will be allocated based on input information obtained through interaction with the user (e.g., +x-axis direction, -x-axis direction, +y-axis direction, - y-axis direction, +z-axis direction, or -z-axis direction), but if there is not enough space to prepare an expanded display area in the determined position or direction, the position of the first display area is moved or expanded. The location where the display area will be provided can be changed, or the direction where the extended display area will be provided can be changed. To this end, the external device 20 may additionally perform interaction with the user.

FIG. 12 is an example diagram of expanding a mirroring screen using a 3D mirroring image in a mirroring system (e.g., mirroring system 1 of FIG. 1) according to an embodiment.

Referring to FIG. 12, an external device included in the mirroring system 1 (e.g., the external device 20 in FIG. 1) uses three-dimensional mirroring data provided from the source device 10 to display the first display area 1210. ) can display a 3D mirroring image.

According to one example, the external device 20 recycles 3D mirroring data previously displayed in the first display area 1210, or as 3D mirroring data is newly provided from the source device 10, the external device 20 displays the first display area 1210. The 3D mirroring data displayed in the display area 1210 can be recycled to display a 3D expanded mirroring image in the extended display area 1220 expanded in the upward direction (+y-axis direction).

According to one example, the external device 20 recycles 3D mirroring data previously displayed in the first display area 1210, or as 3D mirroring data is newly provided from the source device 10, the external device 20 displays the first display area 1210. A 3D expanded mirroring image is displayed in the first expanded display area 1231 corresponding to the expanded display area 1230 expanded to the right (+x-axis direction) by recycling the 3D mirroring data displayed in the display area 1210. can be displayed. In this case, the external device 20 may display a 3D expanded mirroring image that was previously displayed on the first expanded display area 1231 on the second expanded display area 1233.

FIG. 13 shows an extended display area (e.g., of FIG. 6 ) in an external device (e.g., external device 20 of FIG. 1 ) constituting a mirroring system (e.g., mirroring system 1 of FIG. 1 ), according to an embodiment. This is an example diagram providing interaction in the extended display area 640.

Referring to FIG. 13, an external device (e.g., the external device 20 in FIG. 1) displays an extended display screen 1330 on which the previous mirrored image is displayed so that the first display screen 1310 on which the mirrored image is displayed is hidden behind. can be placed. The external device 20 may expand the mirroring image corresponding to each number of times in response to the number of times mirroring has been performed (e.g., TAB number of times) and display it as an independent expanded display screen 1340. For example, expanding the extended display screens 1330, 1341, and 1343 in response to the number of mirroring may be performed by the external device 20 in response to the number of taps of the user through interaction with the user.

When the external device 20 recognizes an input requesting to display the expanded display screen 1320 in the upward direction (+y axis direction) through user interaction (e.g., scrolling), it displays the first display screen ( The expanded display screen 1320 can be exposed by connecting the screens 1310 in the upward direction (+y axis direction).

As described above, the external device 20 controls the display direction of the expanded display screen (e.g., +x-axis direction, -x-axis direction, +y-axis direction, -y-axis direction) according to the user's gesture type through interaction with the user. , +z-axis direction or -z-axis direction) may be determined.

According to one example, the external device 20 may prepare a table in advance of commands to be performed when an interaction is created for each area or range with respect to the first display screen or the extended display screen. When an event is generated in a specific area or range of the first display screen or extended display screen while interacting with the user, the external device 20 defines the specific area or range generated by the event in a table. You can execute actions according to the commands provided. The table may be provided on an external server in addition to the external device 20. The table may also be prepared in software. New commands may be added to the table, depending on the implementation at the time a specific activity or screen is executed.

According to one example, commands according to motion through interaction with the user (e.g., Activity touch, Scroll, etc.) are not provided for each app, and basic commands that can be commonly applied may be applied equally to all apps.

According to one example, it may be possible to implement different methods for each situation using a callback structure in software.

<Table 1> below shows an example of a table defining commands performed for each interaction.

1 area touch - Run the screen using the information (Intent) of activity number 1 and float the activity on the TOP.
- After moving the activity, if the touched area is where a touch event is generated, the corresponding command is executed 2 area touch - Run the screen with the information (Intent) of activity 1 and float the activity on the TOP - After moving the activity, if the touched area is where a touch event is generated, execute the corresponding command 3 area touch - Scroll the page as much as the

According to one example, if the area touched after scrolling is where a touch event is generated, and the page is dynamic when the command is executed, the external device 20 opens the page again and returns it to a different state at the time the touch input is provided. It may have changed. In this case, it can be determined whether the corresponding area is interactable through the difference in pixel values between the image held by the buffer provided in the external device 20 and the newly displayed image.

According to one example, the external device 20 may display a previously displayed mirrored image by moving it to the extended display area as a new mirrored image is mirrored in the first display area. In this case, when the user touches a specific position of the extended mirroring image exposed in the extended display area, the external device 20 can move the extended mirroring image displayed in the extended display area to the first display area and display it. The external device 20 may generate an event by considering that a position corresponding to a specific position touched by the user has been touched when the mirrored image moved to the first display area is displayed in the expanded display area.

FIG. 14 is a history hierarchy tree for managing an extended display area in a mirroring system (eg, mirroring system 1 of FIG. 1) according to an embodiment.

Referring to FIG. 14, an external device (e.g., external device 20 in FIG. 1) that provides an extension area for the mirroring function in the mirroring system 1 interacts with the user (e.g., interaction 25 in FIG. 2). ) An extended display area (e.g., the extended display area 640 of FIG. 6) may be allocated to the display (e.g., the display 600 of FIG. 6). The extended display area 640 allocated to the display 600 may have a hierarchical structure over time. The hierarchical structure is based on the first layer (assessment) 1410, which is the highest layer, the second layer (Group1, Group2, Group3) (1420), and the third layer (Sub_Group11, Sub_Group12, Sub_Group21, Sub_Group22, Sub_Group31 , Sub_Group32) 1430, or a fourth layer (Question) 1440. As an example, the external device 20 may display an extended display area 640 that displays an extended mirroring image on the display 600 as history. It can be managed by mapping to one of the layers constituting the hierarchy. For example, when a specific interaction with a user is performed, hierarchy management can be performed by interaction, such as removing other areas.

FIG. 15 is an example diagram of a mirroring system (eg, the mirroring system 1 of FIG. 1 ) operating an extended display area on its own, according to an embodiment.

Referring to FIG. 15, when software such as an app can be run and operated directly on the external device 20, the external device 20 displays mirroring data (mirroring data) about the app executed on the virtual display of the source device 10. 1510) can be provided from the source device 10. The external device 20 may display a mirroring image on the first display area 1520 using mirroring data 1510 provided from the source device 10. The external device 20 may itself execute the same app as the app executed on the virtual display of the source device 10. The external device 20 may display a screen 1530 according to an app executed by itself on the extended display area 1540.

FIG. 16 is a diagram illustrating an example of expanding a mirroring image in time and space in an external device (eg, the external device 20 of FIG. 1) according to an embodiment.

Referring to FIG. 16 , the external device 20 may display a mirroring image 1620 (hereinafter referred to as 'basic mirroring image 1620') in the first display area 1610. When a future prediction image request event is generated, the external device 20 may copy the mirroring image displayed on the first display area 620 to obtain a duplicate mirroring image 1640. The external device 20 may render a duplicate mirroring image 1640 and display it on the allocated extended display area 1630 so as to obscure the basic mirroring image 1620 displayed on the first display area 1610.

The external device 20 may obtain a future prediction image by performing an operation corresponding to an input provided from the extended display area 1630 displaying the duplicate mirroring image 1640. The future prediction image may be rendered on the external device 20. After the rendering, through interaction with the user, the future prediction image 1660 may be displayed on the extended display area 1650.

FIG. 17 is a control flowchart for expanding a mirroring image in time and space in an external device (eg, the external device 20 of FIG. 1), according to an embodiment.

Referring to FIG. 17, the external device 20 may confirm the occurrence of a future prediction image request event in operation 1711. For example, the future prediction image request event may be generated through the external device 20's interaction with the user. The future predictive image request event is expected to occur next to the basic mirroring image (e.g., basic mirroring image 1620 of FIG. 16) currently displayed in the first display area (e.g., first display area 1610 of FIG. 16). This may be requesting prediction of a mirroring image.

For example, in a specific situation, the external device 20 inputs at least one interaction expected to occur in the future while holding the current mirroring image (e.g., the basic mirroring image 1620 of FIG. 16). In response, an operation according to the event is executed, and the mirroring image obtained as a result of the executed operation (e.g., the future prediction image 1660 of FIG. 16) is displayed in the mirroring display area (e.g., the extended display area 1650 of FIG. 16). )) can be used to visualize it.

More specifically, in operation 1713, the external device 20 may determine whether the app for which a future prediction image is requested is an app capable of linear movement. The fact that the linear movement is possible may mean that the linear movement is free. As an example, apps capable of linear movement may include apps in which mirroring image changes do not occur frequently. The app capable of linear movement may be, for example, an app that provides a weather forecast function.

In operation 1715, the external device 20 may determine whether interaction with the user has not occurred for a certain period of time. The fact that no interaction with the user occurs during the given period of time means that it can be predicted that the possibility of the mirroring image being changed is slim. The fact that no interaction occurs for the certain period of time may correspond to the case where the app running to provide the basic mirroring image 1620 displayed in the first display area 1610 is an app that does not require frequent interaction with the user. .

If the app is capable of linear movement and no interaction has occurred for a certain period of time, the external device 20 copies the basic mirroring image 1620 displayed in the first display area 1610 in operation 1717 and displays the first mirroring image 1620 in the first display area 1610. 1 Rendering can be performed to serve as a screen for the display area 1610. As an example, the external device 20 renders a copied mirroring image (e.g., the duplicate mirroring image 1640 of FIG. 16) and expands it to cover the basic mirroring image 1620 displayed in the first display area 1610. A future prediction image 1660 may be displayed in the display area 1650.

In operation 1719, the external device 20 provides an input to the area displaying the duplicate mirrored image 1640 (e.g., the expanded display area 1630 of FIG. 16) to perform a specified operation, and performs the specified operation. A future prediction image 1660 can be obtained by performing an operation. The step (depth) of performing the operation to obtain the future prediction image 1660 may be performed more than once.

When the future prediction image 1660 is acquired, the external device 20 stores the acquired future prediction image 1660 in a second buffer (e.g., the second buffer 243 in FIG. 3) in operation 1721. You can render using . After the rendering, the future prediction image 1660 may be visually exposed to the user through the extended display area 1650 through interaction with the user.

In operation 1723, the external device 20 removes the shield so that the basic mirroring image 1620 previously displayed in the first display area 1610 can be exposed to the user again.

Due to the above-mentioned reasons, the external device 20 allows the user to experience a future predicted mirroring image by utilizing a wide display area even if the user does not provide a separate function such as previewing the next screen.

FIG. 18 is a block diagram of an electronic device 1801 in a network environment 1800, according to various embodiments.

Referring to FIG. 18, in the network environment 1800, the electronic device 1801 communicates with the electronic device 1802 through a first network 1898 (e.g., a short-range wireless communication network) or a second network 1899. It is possible to communicate with at least one of the electronic device 1804 or the server 1808 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 1801 may communicate with the electronic device 1804 through the server 1808. According to one embodiment, the electronic device 1801 includes a processor 1820, a memory 1830, an input module 1850, an audio output module 1855, a display module 1860, an audio module 1870, and a sensor module ( 1876), interface (1877), connection terminal (1878), haptic module (1879), camera module (1880), power management module (1888), battery (1889), communication module (1890), subscriber identification module (1896) , or may include an antenna module 1897. In some embodiments, at least one of these components (eg, the connection terminal 1878) may be omitted, or one or more other components may be added to the electronic device 1801. In some embodiments, some of these components (e.g., sensor module 1876, camera module 1880, or antenna module 1897) are integrated into one component (e.g., display module 1860). It can be.

The processor 1820, for example, executes software (e.g., program 1840) to operate at least one other component (e.g., hardware or software component) of the electronic device 1801 connected to the processor 1820. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of the data processing or computation, the processor 1820 stores instructions or data received from another component (e.g., the sensor module 1876 or the communication module 1890) in the volatile memory 1832. The commands or data stored in the volatile memory 1832 can be processed, and the resulting data can be stored in the non-volatile memory 1834. According to one embodiment, the processor 1820 includes a main processor 1821 (e.g., a central processing unit or an application processor) or an auxiliary processor 1823 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 1801 includes a main processor 1821 and a auxiliary processor 1823, the auxiliary processor 1823 may be set to use lower power than the main processor 1821 or be specialized for a designated function. You can. The auxiliary processor 1823 may be implemented separately from the main processor 1821 or as part of it.

The auxiliary processor 1823 may, for example, act on behalf of the main processor 1821 while the main processor 1821 is in an inactive (e.g., sleep) state, or while the main processor 1821 is in an active (e.g., application execution) state. ), together with the main processor 1821, at least one of the components of the electronic device 1801 (e.g., the display module 1860, the sensor module 1876, or the communication module 1890) At least some of the functions or states related to can be controlled. According to one embodiment, coprocessor 1823 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 1880 or communication module 1890). there is. According to one embodiment, the auxiliary processor 1823 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 1801 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 1808). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 1830 may store various data used by at least one component (eg, the processor 1820 or the sensor module 1876) of the electronic device 1801. Data may include, for example, input data or output data for software (e.g., program 1840) and instructions related thereto. Memory 1830 may include volatile memory 1832 or non-volatile memory 1834.

The program 1840 may be stored as software in the memory 1830 and may include, for example, an operating system 1842, middleware 1844, or applications 1846.

The input module 1850 may receive commands or data to be used in a component of the electronic device 1801 (e.g., the processor 1820) from outside the electronic device 1801 (e.g., a user). The input module 1850 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 1855 can output sound signals to the outside of the electronic device 1801. The sound output module 1855 may include, for example, a speaker or receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 1860 can visually provide information to the outside of the electronic device 1801 (eg, a user). The display module 1860 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 1860 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 1870 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 1870 acquires sound through the input module 1850, the sound output module 1855, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 1801). Sound may be output through an electronic device 1802 (e.g., speaker or headphone).

The sensor module 1876 detects the operating state (e.g., power or temperature) of the electronic device 1801 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 1876 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 1877 may support one or more designated protocols that can be used to connect the electronic device 1801 directly or wirelessly with an external electronic device (e.g., the electronic device 1802). According to one embodiment, the interface 1877 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 1878 may include a connector through which the electronic device 1801 can be physically connected to an external electronic device (eg, the electronic device 1802). According to one embodiment, the connection terminal 1878 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 1879 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 1879 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 1880 can capture still images and moving images. According to one embodiment, the camera module 1880 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 1888 can manage power supplied to the electronic device 1801. According to one embodiment, the power management module 1888 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

Battery 1889 may supply power to at least one component of electronic device 1801. According to one embodiment, the battery 1889 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

Communication module 1890 provides a direct (e.g., wired) communication channel or wireless communication channel between the electronic device 1801 and an external electronic device (e.g., electronic device 1802, electronic device 1804, or server 1808). It can support establishment and communication through established communication channels. Communication module 1890 operates independently of processor 1820 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 1890 may be a wireless communication module 1892 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1894 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 1898 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 1899 (e.g., legacy It may communicate with an external electronic device 1804 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 1892 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 1896 to communicate within a communication network, such as the first network 1898 or the second network 1899. The electronic device 1801 can be confirmed or authenticated.

The wireless communication module 1892 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 1892 may support high frequency bands (e.g., mmWave bands), for example, to achieve high data rates. The wireless communication module 1892 uses various technologies to secure performance in high frequency bands, such as beamforming, massive MIMO (multiple-input and multiple-output), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 1892 may support various requirements specified in the electronic device 1801, an external electronic device (e.g., electronic device 1804), or a network system (e.g., second network 1899). According to one embodiment, the wireless communication module 1892 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 1897 may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module 1897 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 1897 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for the communication method used in the communication network, such as the first network 1898 or the second network 1899, is connected to the plurality of antennas by, for example, the communication module 1890. can be selected Signals or power may be transmitted or received between the communication module 1890 and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 1897.

According to various embodiments, antenna module 1897 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 1801 and the external electronic device 1804 through the server 1808 connected to the second network 1899. Each of the external electronic devices 1802 or 1804 may be of the same or different type as the electronic device 1801. According to one embodiment, all or part of the operations performed in the electronic device 1801 may be executed in one or more of the external electronic devices 1802, 1804, or 1808. For example, when the electronic device 1801 must perform a function or service automatically or in response to a request from a user or another device, the electronic device 1801 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 1801. The electronic device 1801 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 1801 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 1804 may include an Internet of Things (IoT) device. Server 1808 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, an external electronic device 1804 or a server 1808 may be included in the second network 1899. The electronic device 1801 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

According to one embodiment, a device (e.g., external device 20 of FIG. 1) includes a display module 210 configured to display a mirrored image on a display window 600; a communication module 250 configured to communicate with an external device 10; A sensor module 230 configured to detect a user's gesture and generate a sensing signal corresponding to the detected user's gesture; A memory 240 configured to be allocated one or a plurality of buffer areas 241 and 243; and electrically connected to the display module 210, the communication module 250, or the memory 240, and to control the operation of the display module 210, the communication module 250, or the memory 240. It may include at least one processor 220.

According to one embodiment, the at least one processor 220 receives mirroring data 31 from the external device 10 through the communication module 250 and uses the mirroring data 31 to The display module 210 may be configured to control the mirrored image 650 to be displayed in the first display area 620 allocated in the display window 600.

According to one embodiment, the at least one processor 220 displays additional information in addition to the first display area 620 in the display window 600 according to the user gesture 630 detected by the sensor module 230. The display module 210 may be configured to display the extended mirroring image 660 in at least one allocated extended display area 640 using extended mirroring data 731, 733, and 735.

According to one embodiment, the extended mirroring data 731, 733, and 735 may be obtained from previous mirroring data 731, 733, and 735 that have been used to display the mirrored image 650.

According to one embodiment, at least one processor 220 may be configured to temporarily store the mirroring data received from the external device 10 in the first buffer area 241.

According to one embodiment, when the mirroring image is displayed in the first display area, at least one processor 220 stores the mirroring data temporarily stored in the first buffer area to a second buffer area allocated to the memory. It may be configured to temporarily store the extended mirroring data.

According to one embodiment, when the mirroring image is displayed in the first display area, at least one processor 220 obtains new extended mirroring data from mirroring data temporarily stored in the first buffer area, and stores the new extended mirroring data. It may be configured to temporarily store mirroring data in a second buffer area allocated to the memory.

According to one embodiment, at least one processor 220 acquires original image information included in mirroring data temporarily stored in the first buffer area or rendered image information obtained by rendering the original image information as the new extended mirroring data. It can be configured to do so.

According to one embodiment, the mirroring data may include meta information indicating a link or coordinates and image information to be mirrored.

According to one embodiment, when the at least one processor 220 receives new mirroring data from the external electronic device through the communication module while the extended mirroring image is displayed, the It may be configured to control the display module to reconfigure an extended mirroring image to be displayed in the extended display area based on mirroring screen information used to display the mirrored image.

According to one embodiment, at least one processor 220 moves in the up-down direction (e.g., +y-axis direction or -y-axis direction) or the left-right direction (e.g., +x) in the first display area according to the user gesture. It may be configured to allocate the at least one expanded display area adjacent to each other in an axial direction or -x-axis direction.

According to one embodiment, when the user gesture is detected in the at least one expanded display area, the at least one processor 220 generates input information mapped to the position where the user gesture is detected in the at least one expanded display area. may be configured to identify and control the communication module to transmit the identified input information to the external electronic device.

According to one embodiment, at least one processor 220 is configured to set an area of interest in the display window through eye tracking by the sensor module and allocate the at least one extended display area to the set area of interest. It can be.

According to one embodiment, a display method in an electronic device (e.g., the external device 20 of FIG. 1) includes acquiring mirroring data; The method may include displaying a mirrored image on a first display area allocated on a display using the obtained mirroring data.

According to one embodiment, the display method includes an operation of acquiring extended mirroring data from previous mirroring data that has been used to display the mirrored image, and an additional display area in addition to the first display area on the display according to a user's gesture. The method may include displaying an extended mirroring image using the extended mirroring data in at least one allocated extended display area.

According to one embodiment, the acquiring operation includes receiving the mirroring data from the external electronic device; and temporarily storing the received mirroring data.

The display method according to one embodiment may include temporarily storing the temporarily stored mirroring data as the extended mirroring data when the mirroring image is displayed in the first display area.

According to one embodiment, a display method includes: acquiring new extended mirroring data from the temporarily stored mirroring data when the mirroring image is displayed in the first display area; and temporarily storing the new extended mirroring data.

According to one embodiment, the operation of acquiring the new extended mirroring data includes acquiring original image information included in the temporarily stored mirroring data or rendered image information obtained by rendering the original image information as the new extended mirroring data. It can be included.

According to one embodiment, the mirroring data may include meta information indicating a link or coordinates and image information to be mirrored.

According to one embodiment, a display method includes: acquiring new mirroring data while the extended mirroring image is displayed; and an operation of reconstructing an extended mirroring image to be displayed in the extended display area based on mirroring screen information used to display the mirrored image displayed in the first display area.

According to one embodiment, the operation of displaying the extended mirroring image is performed in the vertical direction (e.g., +y-axis direction or -y-axis direction) or the left-right direction (e.g., +x) in the first display area according to the user gesture. It may include an operation of determining a direction in which to allocate the at least one extended display area (axis direction or -x-axis direction).

According to one embodiment, a display method includes: when the user gesture is detected in the at least one extended display area, identifying input information mapped to a position in the at least one extended display area where the user gesture is detected; and transmitting the identified input information to an external electronic device.

According to one embodiment, the operation of displaying the extended mirroring image includes: setting a region of interest on the display through eye tracking for a user; and allocating the at least one extended display area to the set region of interest.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” When mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are software (e.g., computer) including one or more instructions stored in a storage medium (e.g., memory 140) readable by a machine (e.g., digital device 120). It may be implemented as a program 1840. For example, a processor (e.g., control unit 123) of a device (e.g., electronic device 120) may execute at least one command among one or more commands stored from a storage medium. and execute it. This enables the device to be operated to perform at least one function according to the called at least one instruction. The one or more instructions are code generated by a compiler or by an interpreter. It may contain executable code. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium is tangible. It simply means that it is a device that does not contain signals (e.g. electromagnetic waves), and this term does not distinguish between cases where data is semi-permanently stored in a storage medium and cases where data is stored temporarily.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

Claims (20)

In the device 20,
a display module 210 configured to display a mirrored image on the display window 600;
a communication module 250 configured to communicate with an external device 10;
A sensor module 230 configured to detect a user's gesture and generate a sensing signal corresponding to the detected user's gesture;
A memory 240 configured to be allocated one or a plurality of buffer areas 241 and 243; and
Electrically connected to the display module 210, the communication module 250, or the memory 240, and configured to control the operation of the display module 210, the communication module 250, or the memory 240. Includes at least one processor 220,
The at least one processor 220,
Receiving mirroring data 31 from the external device 10 through the communication module 250,
Controlling the display module 210 to display a mirrored image 650 in the first display area 620 allocated in the display window 600 using the mirroring data 31,
According to the user gesture 630 detected by the sensor module 230, extended mirroring data ( Configured to control the display module 210 to display an extended mirroring image 660 using 731, 733, and 735,
Here, the extended mirroring data (731, 733, 735) is obtained from previous mirroring data (731, 733, 735) that has been used to display the mirrored image (650). According to paragraph 1,
At least one processor 220,
An electronic device configured to temporarily store the mirroring data received from the external device (10) in a first buffer area (241). According to paragraph 2,
At least one processor 220,
When the mirroring image is displayed in the first display area, the electronic device is configured to temporarily store the mirroring data temporarily stored in the first buffer area as the extended mirroring data in a second buffer area allocated to the memory. According to paragraph 2,
At least one processor,
When the mirroring image is displayed in the first display area, new extended mirroring data is obtained from the mirroring data temporarily stored in the first buffer area, and the new extended mirroring data is temporarily stored in a second buffer area allocated to the memory. An electronic device configured to: According to paragraph 4,
The at least one processor,
An electronic device configured to obtain original image information included in mirroring data temporarily stored in the first buffer area or rendered image information obtained by rendering the original image information as the new extended mirroring data. According to paragraph 1,
The mirroring data includes meta information indicating a link or coordinates and image information to be mirrored. According to paragraph 1,
The at least one processor,
When new mirroring data is received from the external electronic device through the communication module while the expanded mirroring image is displayed, the expansion is performed by the mirroring screen information used to display the mirrored image displayed in the first display area. An electronic device that controls the display module to reconfigure an extended mirroring image to be displayed in a display area. According to any one of claims 1 to 7,
The at least one processor,
An electronic device configured to allocate the at least one extended display area adjacent to the first display area in a vertical or left-right direction according to the user gesture. According to any one of claims 1 to 7,
The at least one processor,
When the user gesture is detected in the at least one expanded display area, input information mapped to the position where the user gesture was detected in the at least one expanded display area is identified, and the communication module is controlled to display the identified input information. An electronic device configured to transmit to the external electronic device. According to clause 9,
The at least one processor,
An electronic device configured to set an area of interest in the display window through eye tracking by the sensor module, and allocate the at least one extended display area to the set area of interest. In a display method in an electronic device,
An operation to acquire mirroring data (1013);
An operation of displaying a mirrored image on a first display area allocated on a display using the acquired mirroring data (1015);
Obtaining extended mirroring data from previous mirroring data that has been used to display the mirrored image (1017, 1019); and
A method including an operation (1019) of displaying an extended mirroring image using the extended mirroring data in at least one extended display area additionally allocated in addition to the first display area on the display according to a user's gesture. According to clause 11,
The acquisition operation is,
Receiving the mirroring data from the external electronic device; and
A method comprising temporarily storing the received mirroring data. According to clause 12,
When the mirroring image is displayed in the first display area, the method includes temporarily storing the temporarily stored mirroring data as the extended mirroring data. According to clause 12,
When the mirroring image is displayed in the first display area, acquiring new extended mirroring data from the temporarily stored mirroring data; and
A method comprising temporarily storing the new extended mirroring data. According to clause 14,
The operation of acquiring new extended mirroring data is:
A method comprising acquiring original image information included in the temporarily stored mirroring data or rendered image information obtained by rendering the original image information as the new extended mirroring data. According to clause 11,
The mirroring data includes meta information indicating a link or coordinates and image information to be mirrored. According to clause 11,
Obtaining new mirroring data while the extended mirroring image is displayed; and
A method comprising reconstructing an extended mirroring image to be displayed in the extended display area using mirroring screen information used to display the mirrored image displayed in the first display area. According to any one of claims 11 to 17,
The operation of displaying the extended mirroring image is,
A method comprising determining a direction in which to allocate the at least one extended display area in the first display area according to the user gesture: up and down or left and right. According to any one of claims 11 to 17,
When the user gesture is detected in the at least one expanded display area, identifying input information mapped to a position in the at least one expanded display area where the user gesture is detected; and
A method comprising transmitting the identified input information to an external electronic device. According to clause 19,
The operation of displaying the extended mirroring image is,
An operation of setting a region of interest on the display through eye tracking of a user; and
A method comprising allocating the at least one extended display area to the set region of interest.

Images