KR102614044B1 - Method and apparatus for providing miracast - Google Patents

Abstract

Various embodiments of the present invention disclose a method and device for providing a Miracast service (function) in an electronic device. According to various embodiments of the present invention, an electronic device includes a wireless communication unit for Miracast connection and data transmission and reception, a display for displaying data, and a processor functionally connected to the wireless communication unit and the display, The processor obtains device information from an external electronic device connected to Miracast, determines the resolution and display type of the external electronic device based on the device information, and, if the external electronic device is a vertical display, displays the device on the vertical display. Corresponding data may be generated based on the resolution, and the data may be transmitted to the external electronic device using the wireless communication unit. Various embodiments are possible.

Description

Miracast providing device and method {METHOD AND APPARATUS FOR PROVIDING MIRACAST}

Various embodiments of the present invention disclose a method and device for providing a Miracast service (function) in an electronic device.

With the recent development of digital technology, mobile communication terminals, smart phones, tablet PCs (personal computers), electronic organizers, laptops, PDAs (personal digital assistants), wearable devices, etc. Various types of electronic devices are widely used.

Electronic devices are implemented in the form of multimedia players with complex functions. For example, electronic devices may include call functions such as voice calls and video calls, message sending and receiving functions such as SMS (short message service)/MMS (multimedia message service) and e-mail, wireless (e.g. Bluetooth, Wi-Fi (Wi-Fi, wireless fidelity, etc.) communication function, wired (e.g. HDMI, high definition multimedia interface) communication function, shooting function, broadcast playback function, video playback function, music playback function, Internet function, messenger function, game functions, or social networking service (SNS) functions, etc. may be provided.

Recently, services (or functions) using technology (e.g., Miracast) to wirelessly connect (connect) electronic devices and external electronic devices (e.g., devices capable of display, such as televisions, monitors, etc.) ) is increasing in use. For example, a user can share the screen displayed on the electronic device by displaying it on the display of an external electronic device (e.g., television, monitor, etc.) to which the electronic device is wirelessly connected. These services (or functions) are provided under names such as Miracast, screen mirroring, airplay, or WiDi (wireless display), depending on the manufacturer of the electronic device.

Meanwhile, the current Miracast service (or function) is specialized for cases where the external electronic device (or display) is in landscape orientation. Therefore, the current Miracast does not support portrait orientation displays, or the data of the electronic device may be provided in a rotated manner. For example, in Miracast, data transmitted from an electronic device can be displayed on the display based on the horizontal resolution of the external electronic device (e.g., 1280x1024, 1920x1080, 2560x1440, etc.). Therefore, if the external electronic device has a vertical display, data may be rotated and provided based on the horizontal resolution. For example, data in a portrait layout of an electronic device is rotated 90 degrees (or -90 degrees) through the display and presented in a portrait orientation, or data in a landscape layout of an electronic device is provided in a vertical orientation through the display. It can be rotated 90 degrees (or -90 degrees) and presented in landscape orientation. To correct this, conventionally, through processing (e.g. image processing) to re-correct (e.g. reverse rotation) the rotated data in an external electronic device (e.g. sink device) that receives data, electronic It is intended to have the same layout as the layout displayed on the device. However, this may only be possible if the external electronic device supports functions for image processing.

In various embodiments, an electronic device (e.g., source device) generates shared data (e.g., screen data, playback data) corresponding to the vertical display of an external electronic device (e.g., sink device), and transmits the shared data (e.g., screen data, playback data) to the external electronic device (e.g., sink device). Discloses a method and device for transmission.

In various embodiments, shared data is stored in an electronic device in response to the display type of the external electronic device (e.g., a vertical display with a fixed vertical orientation as a default, a display used by rotating a display with a default horizontal orientation to the vertical orientation). Disclosed is a method and device for generating a.

In various embodiments, when transmitting shared data from an electronic device to an external electronic device, shared data corresponding to the display type of the external electronic device is generated without additional transmission of specific information (e.g., operation mode information or content direction information). Discloses a method and device for transmission.

In various embodiments, an external electronic device does not perform different processing on a user input back channel (UIBC) depending on the operation mode, but generates a control message related to control of the electronic device based on the recognized coordinates. A method and device for transmitting information is disclosed.

An electronic device according to various embodiments of the present invention includes a wireless communication unit for Miracast connection and data transmission and reception, a display for displaying data, and a processor functionally connected to the wireless communication unit and the display, wherein the processor Obtains device information from an external electronic device connected to Miracast, determines the resolution and display type of the external electronic device based on the device information, and, if the external electronic device is a vertical display, displays a device corresponding to the vertical display. Data may be generated based on the resolution, and the data may be transmitted to the external electronic device using the wireless communication unit.

A method of operating an electronic device according to various embodiments of the present invention includes a process of obtaining device information from an external electronic device connected to Miracast, a process of determining the resolution and display type of the external electronic device based on the device information, If the external electronic device is a vertical display, the process may include generating data corresponding to the vertical display based on the resolution and transmitting the data to the external electronic device using the wireless communication unit. .

In order to solve the above problems, various embodiments of the present invention may include a computer-readable recording medium on which a program for executing the method on a processor is recorded.

According to an electronic device and a method of operating the same according to various embodiments, shared data (e.g., screen data, playback data) corresponding to the vertical display of an external electronic device (e.g., sink device) is provided in an electronic device (e.g., source device). ) can be generated and transmitted to an external electronic device. For example, according to various embodiments, the display type of the external electronic device (e.g., a vertical display with a default fixed portrait orientation, a display used by rotating a display with a default horizontal orientation to the portrait orientation) is controlled by the electronic device. In response, shared data can be created. Accordingly, the external electronic device can immediately display the received shared data without performing additional processing on the received shared data (e.g., scaling shared data, rotating shared data, etc.). According to various embodiments, as the electronic device provides shared data according to the resolution and display type of the external electronic device, there is no additional processing of the shared data in the external electronic device, thereby reducing image quality degradation due to scaling or rotation. .

According to various embodiments, when transmitting shared data to an external electronic device, the electronic device generates shared data corresponding to the display type of the external electronic device without additional transmission of specific information (e.g., operation mode information or content direction information). You can transmit it. Accordingly, the electronic device may not perform processing (e.g., encoder change, etc.) according to resolution change, etc., and the resulting current consumption may be reduced. According to various embodiments, as the user input back channel (UIBC) is processed equally according to the operation mode of the external electronic device, user input can be recognized and processed more quickly. Electronic devices according to various embodiments can contribute to improving the usability, convenience, and competitiveness of electronic devices.

1 is a diagram illustrating a network environment including electronic devices according to various embodiments of the present invention.
Figure 2 is a block diagram of an electronic device according to various embodiments of the present invention.
Figure 3 is a block diagram of a program module according to various embodiments of the present invention.
FIG. 4 is a diagram illustrating examples of electronic devices according to various embodiments of the present invention.
FIG. 5 is a diagram illustrating an example of providing a service using Miracast between electronic devices according to various embodiments of the present invention.
FIG. 6 is a flowchart illustrating a method of providing a Miracast service for an electronic device according to various embodiments of the present invention.
Figures 7 and 8 are diagrams to explain an example of an operation scenario of the Miracast service according to various embodiments of the present invention.
Figures 9 and 10 are diagrams to explain another example of an operation scenario of the Miracast service according to various embodiments of the present invention.
FIG. 11 is a flowchart illustrating an operation method for providing a Miracast service of an electronic device according to various embodiments of the present invention.
FIG. 12 is a diagram illustrating an example of an operation for processing user input during Miracast service according to various embodiments of the present invention.
FIG. 13 is a flowchart illustrating a method of processing user input in an electronic device according to various embodiments of the present invention.
FIG. 14 is a flowchart illustrating a method of processing user input in an electronic device according to various embodiments of the present invention.
Figures 15 and 16 are diagrams illustrating operation scenarios for processing user input during Miracast service according to various embodiments of the present invention.

Hereinafter, various embodiments of this document are described with reference to the attached drawings. The embodiments and terms used herein are not intended to limit the technology described in this document to a specific embodiment, and should be understood to include various changes, equivalents, and/or substitutes for the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar components. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. In this document, expressions such as “A or B” or “at least one of A and/or B” may include all possible combinations of the items listed together. Expressions such as “first,” “second,” “first,” or “second” can modify the elements in question, regardless of order or importance, and can be used to distinguish one element from another. However, it does not limit the components. When a component (e.g., a first) component is said to be "connected (functionally or communicatively)" or "connected" to another (e.g., second) component, it means that the component is connected to the other component. It may be connected directly to a component or may be connected through another component (e.g., a third component).

In this document, "configured to" means "suitable for," "having the ability to," depending on the situation, for example, in terms of hardware or software. ", "changed to," "made to," "capable of," or "designed to." In some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., central processing unit (CPU) or application processor (AP)) capable of performing the corresponding operations.

Electronic devices according to various embodiments of the present document include, for example, smartphones, tablet PCs, mobile phones, video phones, e-book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, and personal digital assistants (PDAs). assistant), a portable multimedia player (PMP), an MP3 player, a medical device, a camera, or a wearable device. Wearable devices may be accessory (e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-devices (HMDs)), fabric or clothing-integrated (e.g., electronic clothing), The electronic device may include at least one of a body-attached circuit (e.g., a skin pad or a tattoo), or an implantable circuit. In some embodiments, the electronic device may include, for example, a television, a DVD, or a television. (digital video disk) player, audio, refrigerator, air conditioner, vacuum cleaner, oven, microwave, washing machine, air purifier, set-top box, home automation control panel, security control panel, media box (e.g. Samsung HomeSync ^TM , Apple TV ^TM , or Google TV ^TM ), a game console (e.g., Xbox ^TM , PlayStation ^TM ), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

In another embodiment, the electronic device may include various medical devices (e.g., various portable medical measurement devices (e.g., blood glucose meters, heart rate monitors, blood pressure monitors, or temperature monitors), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), etc. ), CT (computed tomography), imaging device, or ultrasonic device, etc.), navigation device, global navigation satellite system (GNSS), EDR (event data recorder), FDR (flight data recorder), automobile infotainment device, marine electronics Equipment (e.g. marine navigation systems, gyro compasses, etc.), avionics, security devices, head units for vehicles, industrial or domestic robots, drones, and automated teller machines (ATMs) in financial institutions. , a store's point of sales (POS), or Internet of Things (IoT) devices (e.g. light bulbs, various sensors, sprinkler devices, fire alarms, thermostats, street lights, toasters, exercise equipment, hot water tanks, heaters, A boiler, etc.) may be included. According to some embodiments, the electronic device may be a piece of furniture, a building/structure, or a vehicle, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (e.g., water, electrical, It may include at least one of gas, radio wave measuring equipment, etc.). In various embodiments, the electronic device may be flexible, or may be a combination of two or more of the various devices described above. Electronic devices according to embodiments of this document are not limited to the above-mentioned devices. In this document, the term user may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

FIG. 1 is a diagram illustrating a network environment including electronic devices according to various embodiments.

1, an electronic device 101 within a network environment 100, in various embodiments, is described. The electronic device 101 may include a bus 110, a processor 120, a memory 130, an input/output interface 150, a display 160, and a communication interface 170. In some embodiments, the electronic device 101 may omit at least one of the components or may additionally include another component.

The bus 110 connects the components 110 to 170 to each other and may include circuitry that transfers communication (eg, control messages or data) between the components.

The processor 120 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 may, for example, perform operations or data processing related to control and/or communication of at least one other component of the electronic device 101. Processing (or control) operations of the processor 120 according to various embodiments will be described in detail with reference to the drawings described later.

Memory 130 may include volatile memory and/or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to one embodiment, memory 130 may store software and/or program 140. The program 140 includes, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and/or an application program. (or “application”) 147, etc. may be included. At least a portion of the kernel 141, middleware 143, or API 145 may be referred to as an operating system (OS).

The memory 130 may store one or more programs executed by the processor 120 and may also perform a function for temporary storage of input/output data. According to various embodiments, the memory 130 is responsible for storing acquired data, and data acquired in real time can be stored in a temporary storage device (e.g., a buffer), and data confirmed to be stored can be stored for a long time. It can be saved to any available storage device. The memory 130 may include a computer-readable recording medium on which a program for executing a method according to various embodiments of the present invention on the processor 120 is recorded.

Kernel 141 may, for example, provide system resources (e.g., middleware 143, API 145, or application program 147) used to execute operations or functions implemented in other programs (e.g., : Bus 110, processor 120, or memory 130, etc.) can be controlled or managed. In addition, the kernel 141 provides an interface for controlling or managing system resources by accessing individual components of the electronic device 101 in the middleware 143, API 145, or application program 147. You can.

The middleware 143 may, for example, perform an intermediary role so that the API 145 or the application program 147 can communicate with the kernel 141 to exchange data. Additionally, the middleware 143 may process one or more work requests received from the application program 147 according to priority. For example, the middleware 143 may use system resources (e.g., bus 110, processor 120, or memory 130, etc.) of the electronic device 101 for at least one of the application programs 147. Priority may be assigned and the one or more work requests may be processed. The API 145 is an interface for the application 147 to control functions provided by the kernel 141 or middleware 143, for example, file control, window control, image processing, or It may include at least one interface or function (e.g., command) for character control, etc.

For example, the input/output interface 150 transmits commands or data input from a user or an external device to other component(s) of the electronic device 101, or to other component(s) of the electronic device 101. ) can be output to the user or an external device. For example, wired/wireless headphone port, external charger port, wired/wireless data port, memory card port, audio input/output port, video input/output port, earphone port. etc. may be included in the input/output interface 150.

The display 160 may be, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or an active matrix OLED (AMOLED). ), micro-electromechanical systems (MEMS) displays, or electronic paper displays. For example, the display 160 may display various contents (e.g., text, images, videos, icons, and/or symbols, etc.) to the user. The display 160 may include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a part of the user's body.

The display 160 may, for example, show visual output to the user. Visual output can appear in the form of text, graphics, video, and combinations thereof. The display 160 can display (output) various information processed by the electronic device 101. For example, the display 160 may display a user interface (UI) or a graphical user interface (GUI) related to the use of the electronic device.

According to various embodiments, the display 160 may display various UIs or GUIs related to operations performed by the electronic device 101. In the electronic device 101 according to various embodiments, various examples of screens provided based on a user interface will be described with reference to the drawings described later.

The communication interface 170, for example, establishes communication between the electronic device 101 and an external device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106). You can. For example, the communication interface 170 may be connected to the network 162 through wireless or wired communication and communicate with an external device (eg, the second external electronic device 104 or the server 106).

Wireless communications include, for example, long term evolution (LTE), LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), and wireless broadband (WiBro). , or GSM (global system for mobile communications), etc. may include cellular communication using at least one of the following. According to one embodiment, wireless communication includes, for example, WiFi (wireless fidelity), WiGig (wireless gigabit alliance), Bluetooth, Bluetooth low energy (BLE), Zigbee, and NFC (near). It may include at least one of field communication, magnetic secure transmission, radio frequency (RF), or body area network (BAN). According to one embodiment, wireless communications may include GNSS. GNSS may be, for example, global positioning system (GPS), global navigation satellite system (Glonass), Beidou Navigation Satellite System (hereinafter “Beidou”), or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, “GPS” may be used interchangeably with “GNSS.”

Wired communications include, for example, universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), power line communication, or plain old telephone service (POTS). It can contain at least one.

Network 162 is a telecommunications network, for example, at least one of a computer network (e.g., a local area network (LAN) or a wide area network (WAN)), the Internet, or a telephone network. It can be included.

Each of the first external electronic device 102 and the second external electronic device 104 may be the same or different type of device from the electronic device 101. According to various embodiments, all or part of the operations performed in the electronic device 101 may be executed in one or more electronic devices (eg, the electronic devices 102 and 104, or the server 106). According to one embodiment, when the electronic device 101 is to perform a certain function or service automatically or upon request, the electronic device 101 performs the function or service instead of or in addition to executing the function or service on its own. At least some of the related functions may be requested from another device (eg, the electronic devices 102 and 104, or the server 106). Another electronic device (e.g., electronic device 102, 104, or server 106) may execute the requested function or additional function and transmit the result to electronic device 101. The electronic device 101 may process the received results as is or additionally to provide the requested function or service. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology can be used.

2 is a block diagram of an electronic device according to various embodiments.

The electronic device 201 may include, for example, all or part of the electronic device 101 shown in FIG. 1 . The electronic device 201 includes one or more processors (e.g., AP) 210, a communication module 220, a subscriber identification module 224, a memory 230, a sensor module 240, an input device 250, and a display ( 260), an interface 270, an audio module 280, a camera module 291, a power management module 295, a battery 296, an indicator 297, and a motor 298. In various embodiments, the electronic device 201 may be implemented with more or fewer configurations than the configurations shown in FIG. 2 because the configurations shown in FIG. 2 are not essential. For example, the electronic device 201 according to various embodiments may not include some components depending on its type. According to various embodiments, the components of the electronic device 201 described above may be mounted on the housing (or bezel, main body) of the electronic device 201, or may be formed outside the housing.

The processor 210, for example, can run an operating system or an application program to control a number of hardware or software components connected to the processor 210, and can perform various data processing and calculations. The processor 210 may be implemented, for example, as a system on chip (SoC). According to one embodiment, the processor 210 may further include a graphic processing unit (GPU) and/or an image signal processor (ISP). The processor 210 may include at least some of the components shown in FIG. 2 (eg, the cellular module 221). The processor 210 may load and process commands or data received from at least one of the other components (eg, non-volatile memory) into the volatile memory, and store the resulting data in the non-volatile memory.

In various embodiments, the processor 210 may control the overall operation of the electronic device 201. In various embodiments, processor 210 may include one or more processors. For example, the processor 210 includes a communication processor (CP), an application processor (AP), an interface (e.g., general purpose input/output (GPIO)), or internal memory as separate components, or one or more It can be integrated into an integrated circuit. According to one embodiment, the application processor may execute various software programs to perform various functions for the electronic device 201, and the communication processor may perform processing and control for voice communication and data communication. It can be done. The processor 210 may be responsible for executing a specific software module (eg, an instruction set) stored in the memory 230 and performing various specific functions corresponding to the module.

In various embodiments, the processor 210 may control the operation of hardware modules such as the audio module 280, the interface 270, the display 260, the camera module 291, and the communication module 220. According to various embodiments, the processor 210 may be electrically connected to the display 260 and the memory 230 of the electronic device 201.

According to various embodiments, the processor 210 may perform an operation of obtaining device information from the external electronic device 500 connected to Miracast, an operation of determining the resolution and display type of the external electronic device 500 based on the device information, and an operation of determining the resolution and display type of the external electronic device 500 based on the device information. If the electronic device 500 is a vertical display, the operation of generating data corresponding to the vertical display based on the resolution and the operation of transmitting the data to the external electronic device 500 using a wireless communication unit can be processed. . According to various embodiments, the processor 210 determines the optimal resolution common to the external electronic device 500 based on device information, determines the display type of the external electronic device 500, and determines whether the display type is a vertical display. In this case, a vertical virtual display with a determined resolution can be created, and if the display type is a horizontal display, operations related to creating a horizontal virtual display with a determined resolution can be processed. According to various embodiments, the processor 210 may process operations related to generating data corresponding to the resolution and display type of the external electronic device 500 using a virtual display.

Processing (or control) operations of the processor 210 according to various embodiments will be described in detail with reference to the drawings described later.

For example, the communication module 220 may have the same or similar configuration as the communication interface 170 shown in FIG. 1. The communication module 220 may include, for example, a cellular module 221, a WiFi module 223, a Bluetooth module 225, a GNSS module 227, an NFC module 228, and an RF module 229. there is. Although not shown, the communication module 220 may further include, for example, a WiGig module (not shown). According to one embodiment, the WiFi module 223 and the WiGig module (not shown) may be integrated and implemented in the form of a single chip.

For example, the cellular module 221 may provide voice calls, video calls, text services, or Internet services through a communication network. According to one embodiment, the cellular module 221 may use a subscriber identification module (e.g., a subscriber identification module (SIM) card) 224 to distinguish and authenticate the electronic device 201 within a communication network. . According to one embodiment, the cellular module 221 may perform at least some of the functions that the processor 210 can provide. According to one embodiment, the cellular module 221 may include a communication processor (CP). According to some embodiments, at least some (e.g., two or more) of the cellular module 221, WiFi module 223, Bluetooth module 225, GNSS module 227, or NFC module 228 include one IC ( It may be included within an integrated chip (IC) or IC package.

For example, the RF module 229 may transmit and receive communication signals (eg, RF signals). The RF module 229 may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 221, WiFi module 223, Bluetooth module 225, GNSS module 227, or NFC module 228 transmits and receives RF signals through a separate RF module. You can.

For example, the WiFi module 223 may represent a module for connecting to the wireless Internet and forming a wireless LAN link with an external device (e.g., another electronic device 102 or server 106, etc.). The WiFi module 223 may be built into or external to the electronic device 201. Wireless Internet technologies may include WiFi, WiGig, Wibro, WiMax (world interoperability for microwave access), HSDPA (high speed downlink packet access), or mmWave (millimeter Wave). The WiFi module 223 is connected directly to an electronic device (e.g., peer-to-peer (P2P) connection) or is externally connected through a network (e.g., a wireless Internet network) (e.g., network 162). By linking with a device (e.g., another electronic device 104, etc.), various data of the electronic device 201 can be transmitted to or received from the outside. The WiFi module 223 may remain on at all times or may be turned on/off depending on the settings of the electronic device or user input.

The Bluetooth module 225 and the NFC module 228 may represent, for example, a short-range communication module for performing short-range communication. Short-range communication technologies may include Bluetooth, Bluetooth low energy (BLE), radio frequency identification (RFID), infrared communication (IrDA), ultra wideband (UWB), Zigbee, or NFC. The short-range communication module links the electronic device 201 with an external device (e.g., another electronic device 102, etc.) connected through a network (e.g., a short-range communication network) to transmit various data of the electronic device 201 to an external device. It can be sent to or received from a device. The short-range communication module (e.g., the Bluetooth module 225 and the NFC module 228) may remain on at all times or may be turned on/off depending on the settings of the electronic device 201 or user input.

Subscriber identity module 224 may include, for example, a card or embedded SIM that includes a subscriber identity module and may include unique identification information (e.g., integrated circuit card identifier (ICCID)) or subscriber information (e.g., IMSI). (international mobile subscriber identity)).

The memory 230 (eg, memory 130) may include, for example, an internal memory 232 or an external memory 234. The built-in memory 232 may include, for example, volatile memory (e.g., dynamic random access memory (DRAM), synchronous RAM (SRAM), or synchronous dynamic RAM (SDRAM), etc.), non-volatile memory (e.g., OTPROM), (one time programmable ROM (read only memory)), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically EPROM (EEPROM), mask ROM, flash ROM, flash memory, hard drive, or solid state drive (SSD) (solid state drive). The external memory 234 may include a flash drive, for example, compact flash (CF), secure digital (SD), Micro-SD, Mini- It may include SD, xD (extreme digital), MMC (multi-media card), memory stick, etc. The external memory 234 may be functionally or physically connected to the electronic device 201 through various interfaces.

In various embodiments, the memory 230 allows the processor 210 to obtain device information from the external electronic device 500 connected to Miracast and determine the resolution and display type of the external electronic device 500 based on the device information. And, if the external electronic device 500 is a vertical display, data corresponding to the vertical display is generated based on the resolution and the data is transmitted to the external electronic device 500 using a wireless communication unit. Or more programs, data or instructions can be stored. According to various embodiments, the memory 230 determines the optimal resolution common to the external electronic device 500 and the display type of the external electronic device 500 based on device information, and determines the display type of the external electronic device 500 when the display type is a vertical display. , It is possible to create a vertical virtual display with a determined resolution and, when the display type is a horizontal display, to store one or more programs, data or instructions related to creating a horizontal virtual display with a determined resolution. there is. According to various embodiments, the memory 230 stores one or more programs, data, or instructions related to generating data corresponding to the resolution and display type of the external electronic device 500 using a virtual display. You can.

The memory 230 may include expanded memory (eg, external memory 234) or internal memory (eg, built-in memory 232). The electronic device 201 may operate in connection with web storage that performs a storage function of the memory 230 on the Internet.

Memory 230 may store one or more software (or software modules). For example, the software component may include an operating system software module, a communications software module, a graphics software module, a user interface software module, a moving picture experts group (MPEG) module, a camera software module, or one or more application software modules. may include. Additionally, since a module, which is a software component, can be expressed as a set of instructions, the module is also expressed as an instruction set. Modules are also expressed as programs. In various embodiments of the present invention, the memory 230 may include additional modules (instructions) in addition to the modules described above. Or, if necessary, some modules (commands) may not be used.

Operating system software modules may include various software components that control general system operation. Control of these general system operations may mean, for example, memory management and control, storage hardware (device) control and management, or power control and management. Additionally, operating system software modules can also perform functions that facilitate communication between various hardware (devices) and software components (modules).

The communication software module may enable communication with other electronic devices such as wearable devices, smartphones, computers, servers, refrigerators, televisions, monitors, or portable terminals through the communication module 220 or interface 270. Additionally, the communication software module may be configured with a protocol structure corresponding to the corresponding communication method.

The graphics software module may include various software components for providing and displaying graphics on the display 260. In various embodiments, the term graphics may be used to include text, web pages, icons, digital images, video, animation, etc. there is.

A user interface software module may include various software components related to a user interface (UI). For example, it may include information about how the state of the user interface is changed or under what conditions the change in the state of the user interface occurs.

The MPEG module may include software components that enable digital content (e.g. video, audio) related processes and functions (e.g. creation, playback, distribution and transmission of content, etc.).

A camera software module may include camera-related software components that enable camera-related processes and functions.

The application module is a web browser including a rendering engine, email, instant message, word processing, keyboard emulation, and address book. , touch list, widget, digital right management (DRM), iris scan, context cognition, voice recognition, position determination function It may include determining function, location based service, etc. According to various embodiments of the present invention, the application module includes instructions for displaying and sharing the screen (or content) displayed on the electronic device 201 through the display of an external electronic device to which the electronic device 210 is wirelessly connected. can do.

For example, the sensor module 240 may measure a physical quantity or sense the operating state of the electronic device 201 and convert the measured or sensed information into an electrical signal. The sensor module 240 includes, for example, a gesture sensor 240A, a gyro sensor 240B, a barometer sensor 240C, and a magnetic sensor 240D. , acceleration sensor (240E), grip sensor (240F), proximity sensor (240G), color sensor (240H) (e.g. RGB (red, green, blue) ) sensor), a medical sensor (240I), a temperature-humidity sensor (240J), an illuminance sensor (240K), or a UV (ultra violet) sensor (240M). It can contain one. Additionally or alternatively, the sensor module 240 may include, for example, an olfactory (e-nose) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor. , it may include an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an iris scan sensor, and/or a fingerprint sensor. The sensor module 240 may further include a control circuit for controlling at least one sensor included therein. In some embodiments, the electronic device 201 further includes a processor configured to control the sensor module 240, either as part of the processor 210 or separately, while the processor 210 is in a sleep state, The sensor module 240 can be controlled.

The input device 250 may include, for example, a touch panel 252, a (digital) pen sensor 254, a key 256, or an ultrasonic input device 258. The touch panel 252 may use at least one of, for example, a capacitive type, a resistive type, an infrared type, or an ultrasonic type. Additionally, the touch panel 252 may further include a control circuit. The touch panel 252 may further include a tactile layer to provide a tactile response to the user. The (digital) pen sensor 254 may be, for example, part of a touch panel or may include a separate recognition sheet. Keys 256 may include, for example, physical buttons, optical keys, or keypads. The ultrasonic input device 258 can detect ultrasonic waves generated from the input tool through the microphone 288 and check data corresponding to the detected ultrasonic waves. According to various embodiments, the input device 250 may include an electronic pen. According to various embodiments, the input device 250 may be implemented to receive force touch input.

Display 260 (e.g., display 160) may include a panel 262, a holographic device 264, a projector 266, and/or control circuitry for controlling them.

Panel 262 may be implemented as flexible, transparent, or wearable, for example. The panel 262 may be composed of a touch panel 252 and one or more modules. According to one embodiment, the panel 262 may include a pressure sensor (or force sensor) that can measure the intensity of pressure in response to the user's touch. The pressure sensor may be implemented integrally with the touch panel 252, or may be implemented as one or more sensors separate from the touch panel 252. The panel 262 may be seated on the display 260 and may detect a user input that touches or approaches the surface of the display 260. The user input may include a touch input or proximity input based on at least one of single-touch, multi-touch, hovering, or air gesture. In various embodiments, the panel 262 may receive a user input to initiate an operation related to the use of the electronic device 201 and may generate an input signal according to the user input. The panel 262 may be configured to convert a change in pressure applied to a specific part of the display 260 or a change in capacitance occurring in a specific part of the display 260 into an electrical input signal. The panel 262 can detect the location and area where an input tool (eg, a user's finger, an electronic pen, etc.) touches or approaches the surface of the display 260. Additionally, the panel 262 may be implemented to detect pressure upon touch (eg, force touch) depending on the applied touch method.

The hologram device 264 can display a three-dimensional image in the air using light interference. The projector 266 can display an image by projecting light onto the screen. The screen may be located, for example, inside or outside the electronic device 201.

The interface 270 may include, for example, HDMI 272, USB 274, optical interface 276, or D-subminiature (D-sub) 278. Interface 270 may be included in, for example, communication interface 170 shown in FIG. 1 . Additionally or alternatively, the interface 270 may include, for example, a mobile high-definition link (MHL) interface, a SD card/multi-media card (MMC) interface, or an infrared data association (IrDA) standard interface. there is.

The interface 270 can receive data from another electronic device or receive power and transmit it to each component within the electronic device 201. The interface 270 may allow data inside the electronic device 201 to be transmitted to another electronic device. For example, wired/wireless headphone port, external charger port, wired/wireless data port, memory card port, audio input/output port, video input/output port, earphone port. etc. may be included in the interface 270.

The audio module 280 can, for example, convert sound and electrical signals in two directions. At least some components of the audio module 280 may be included in, for example, the input/output interface 145 shown in FIG. 1 . The audio module 280 may process sound information input or output through, for example, a speaker 282, a receiver 284, an earphone 286, or a microphone 288. The audio module 280 transmits the audio signal input from the processor 210 to an output device (e.g., speaker 282, receiver 284, or earphone 286) and an input device (e.g., microphone 288). It may perform a function of transmitting audio signals such as voice input from to the processor 210. The audio module 280 converts voice/sound data into audible sound through an output device under the control of the processor 210 and outputs it, and converts audio signals such as voice received from the input device into digital signals to be transmitted to the processor 210. It can be delivered to.

The speaker 282 or the receiver 284 may output audio data received from the communication module 220 or stored in the memory 230. The speaker 282 or the receiver 284 may output sound signals related to various operations (functions) performed in the electronic device. The microphone 288 can receive an external acoustic signal and process it into electrical voice data. Various noise reduction algorithms may be implemented in the microphone 288 to remove noise generated in the process of receiving an external acoustic signal. The microphone 288 may be responsible for inputting audio streaming, such as voice commands.

The camera module 291 is, for example, a device capable of shooting still images and moving images, and according to one embodiment, it includes one or more image sensors (e.g., a front sensor or a rear sensor), a lens, and an image signal processor (ISP). , or may include a flash (e.g., LED or xenon lamp, etc.).

According to various embodiments, the camera module 291 may, for example, use a first camera (e.g., a color (RGB) camera) and depth information (e.g., location information of a subject, distance, etc.) to obtain color information. It may include a second camera (e.g., an infrared (IR) camera) for acquiring information. For example, a first camera (eg, a color camera) can capture a color image of a subject by converting light input from the outside into an image signal. The camera module 291 may include an image sensor. The image sensor may be implemented as a charged coupled device (CCD) or complementary metal-oxide semiconductor (CMOS). According to one embodiment, the first camera may be a front camera provided on the front of the electronic device 201. According to various embodiments, the front camera may be replaced by a second camera and may not be provided on the front of the electronic device 201. According to various embodiments, the first camera may be placed together with the second camera on the front of the electronic device 201. According to one embodiment, the first camera may be a rear camera provided on the rear of the electronic device 201. According to one embodiment, the first camera may include both a front camera and a rear camera provided on the front and rear of the electronic device 201, respectively.

The power management module 295 may manage power of the electronic device 201, for example. According to one embodiment, the power management module 295 may include a power management integrated circuit (PMIC), a charging IC, a battery, or a fuel gauge. PMIC may have wired and/or wireless charging methods. The wireless charging method includes, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic wave method, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonance circuit, or a rectifier. there is. The battery gauge may measure, for example, the remaining amount of the battery 296, voltage, current, or temperature during charging. Battery 296 may include, for example, a rechargeable battery and/or a solar battery.

The indicator 297 may display a specific state of the electronic device 201 or a part thereof (eg, the processor 210), such as a booting state, a message state, or a charging state. The motor 298 can convert electrical signals into mechanical vibration and generate vibration or haptic effects. The electronic device 201 is, ^for example, a mobile TV support device (e.g. : GPU) may be included.

Each of the components described in this document may be composed of one or more components, and the names of the components may vary depending on the type of electronic device. In various embodiments, an electronic device (e.g., the electronic devices 101 and 201) omits some components, further includes additional components, or combines some of the components to form a single entity. , the functions of the corresponding components before combination can be performed identically.

Figure 3 is a block diagram of a program module according to various embodiments.

According to one embodiment, the program module 310 (e.g., program 140) is an operating system that controls resources related to electronic devices (e.g., electronic devices 101, 201) and/or various operating systems running on the operating system. It may include an application (e.g., an application program 147). The operating system may include, for example, Android ^TM , iOS ^TM , Windows ^TM , Symbian ^TM , Tizen ^TM , or Bada ^TM .

Referring to FIG. 3, the program module 310 includes a kernel 320 (e.g., kernel 141), middleware 330 (e.g., middleware 143), and API 360 (e.g., API 145). , and/or an application 370 (e.g., an application program 147). At least a portion of the program module 310 may be preloaded on the electronic device or downloaded from an external electronic device (eg, the electronic devices 102 and 104, the server 106, etc.).

The kernel 320 may include, for example, a system resource manager 321 and/or a device driver 323. The system resource manager 321 may control, allocate, or retrieve system resources. According to one embodiment, the system resource manager 321 may include a process management unit, a memory management unit, or a file system management unit. The device driver 323 may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WiFi driver, an audio driver, or an inter-process communication (IPC) driver. . For example, the middleware 330 provides functions commonly required by the application 370 or provides various functions through the API 360 so that the application 370 can use limited system resources inside the electronic device. It can be provided as an application (370).

According to one embodiment, the middleware 330 includes a runtime library 335, an application manager 341, a window manager 342, and a multimedia manager 343. , resource manager (344), power manager (345), database manager (346), package manager (347), connectivity manager (348) , may include at least one of a notification manager (349), a location manager (350), a graphic manager (351), or a security manager (352). there is.

The runtime library 335 may include, for example, a library module used by a compiler to add new functions through a programming language while the application 370 is running. The runtime library 335 may perform input/output management, memory management, or arithmetic function processing.

The application manager 341 may, for example, manage the life cycle of the application 370. The window manager 342 can manage graphical user interface (GUI) resources used on the screen. The multimedia manager 343 can determine the format required to play media files and encode or decode the media files using a codec suitable for the format. The resource manager 344 may manage the source code or memory space of the application 370. The power manager 345 may, for example, manage battery capacity or power and provide power information necessary for the operation of the electronic device. According to one embodiment, the power manager 345 may be linked to a basic input/output system (BIOS). The database manager 346 may create, search, or change a database to be used in the application 370, for example. The package manager 347 can manage the installation or update of applications distributed in the form of package files.

Connectivity manager 348 may manage wireless connections, for example. The notification manager 349 may provide events such as arrival messages, appointments, and proximity notifications to the user, for example. The location manager 350 may, for example, manage location information of the electronic device. The graphics manager 351 may, for example, manage graphic effects to be provided to users or user interfaces related thereto. Security manager 352 may provide, for example, system security or user authentication.

According to one embodiment, the middleware 330 may include a telephony manager for managing the voice or video call function of the electronic device or a middleware module that can form a combination of the functions of the above-described components. According to one embodiment, the middleware 330 may provide specialized modules for each type of operating system. The middleware 330 may dynamically delete some existing components or add new components.

The API 360 is, for example, a set of API programming functions and may be provided in different configurations depending on the operating system. For example, in the case of Android or iOS, one API set can be provided for each platform, and in the case of Tizen, two or more API sets can be provided for each platform.

The application 370 includes, for example, home 371, dialer 372, SMS/MMS (373), instant message (IM) 374, browser 375, camera 376, and alarm 377. , contact (378), voice dial (379), email (380), calendar (381), media player (382), album (383), watch (384), health care (e.g., measuring the amount of exercise or blood sugar, etc.) , or may include an application that provides environmental information (e.g., atmospheric pressure, humidity, or temperature information).

According to one embodiment, the application 370 may include an information exchange application that can support information exchange between an electronic device and an external electronic device. The information exchange application may include, for example, a notification relay application for delivering specific information to an external electronic device, or a device management application for managing an external electronic device. For example, a notification relay application may transmit notification information generated by another application of an electronic device to an external electronic device, or may receive notification information from an external electronic device and provide it to the user. A device management application may, for example, control the functionality of an external electronic device to communicate with the electronic device, e.g. turn-on/turn-off the external electronic device itself (or some of its components). ) or adjusting the brightness (or resolution) of the display), or installing, deleting, or updating applications running on an external electronic device. According to one embodiment, the application 370 may include an application designated according to the properties of the external electronic device (eg, a health management application for a mobile medical device).

According to one embodiment, the application 370 may include an application received from an external electronic device. At least a portion of the program module 310 may be implemented (e.g., executed) in software, firmware, hardware (e.g., processor 210), or a combination of at least two of these, and may be a module for performing one or more functions; May contain programs, routines, instruction sets, or processes.

As used in this document, the term “module” includes a unit consisting of hardware, software, or firmware, such as logic, logic block, component, or circuit. ) can be used interchangeably with terms such as A “module” may be an integrated part, a minimum unit that performs one or more functions, or a part thereof. A “module” may be implemented mechanically or electronically, for example, in application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), known or hereafter developed, that perform certain operations. May include programmable logic devices. At least a portion of the device (e.g., modules or functions) or method (e.g., operations) according to various embodiments includes instructions stored in a computer-readable storage medium (e.g., memory 130) in the form of a program module. It can be implemented as: When the instruction is executed by a processor (eg, processor 120), the processor may perform the function corresponding to the instruction.

Computer-readable recording media include hard disks, floppy disks, magnetic media (e.g. magnetic tape), optical recording media (e.g. CD-ROM (compact disc read only memory), DVD (digital versatile disc), magneto-optical media (e.g., floptical disk), internal memory, etc. Instructions may be code generated by a compiler or executed by an interpreter. A module or program module according to various embodiments may include at least one of the above-described components, some of them may be omitted, or may further include other components. According to the examples, the operations performed by the module, program module or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some of the operations may be executed in a different order, omitted, or other operations may be performed sequentially, in parallel, iteratively or heuristically. can be added

According to various embodiments, the recording medium may include a computer-readable recording medium recording a program for executing various methods described later on the processors 120 and 210.

In various embodiments, an electronic device may include any device that uses one or more of various processors, such as an AP, CP, GPU, and CPU. For example, electronic devices according to various embodiments may include information and communication devices, multimedia devices, wearable devices, IoT devices, or various other devices corresponding to these devices.

Hereinafter, operation methods and devices according to various embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be noted that the various embodiments of the present invention are not restricted or limited by the content described below, and can be applied to various embodiments based on the examples below. In various embodiments of the present invention described below, a hardware approach method is explained as an example. However, since various embodiments of the present invention include technology using both hardware and software, various embodiments of the present invention do not exclude software-based approaches.

FIG. 4 is a diagram illustrating examples of electronic devices according to various embodiments of the present invention.

As shown in FIG. 4, a system according to various embodiments of the present invention includes an electronic device 400 (e.g., a first electronic device or source device) and at least one external electronic device 500 (e.g., : may include a second electronic device or sink device).

In various embodiments, the electronic device 400 includes a display 417, a housing (or main body) 450 in which the display 417 is seated and fastened, and a function of the electronic device 400 formed on the housing 450. It may be configured to include additional devices for performance, etc. In various embodiments, the additional device may include a first speaker 401, a second speaker 403, a microphone 405, a sensor (e.g., a front camera 407, an illumination sensor 409, etc.), and a communication interface (e.g., charging Alternatively, it may include a data input/output port 411, an audio input/output port 413, etc.), a button 415, etc.

In various embodiments, the display 417 may include a flat display or a curved display (or bended display) that can be bent, bent, or rolled without damage through a thin, flexible substrate like paper. . The curved display may be fastened to the housing 450 (or bezel, main body) to maintain a curved shape. In various embodiments, the electronic device 400 has a curved display-like shape, as well as a flexible display-like shape. It may also be implemented as a display device that can be freely opened and opened. In various embodiments, the display 417 may be a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), or an active OLED ( In AMOLED, active matrix OLED, etc., the glass substrate surrounding the liquid crystal can be replaced with a plastic film to provide flexibility to fold and unfold. In various embodiments, the display 417 is located on at least one side of the electronic device 400. (side) (e.g., at least one of the left, right, upper, and lower sides) and has a radius of curvature at which the curved display can operate (e.g., 5 cm, 1 cm, 7.5 mm, 5 mm, It can be folded to less than 4 mm and fastened to the side of the housing 450.

In various embodiments, the electronic device 400 may recognize the user's fingerprint using a fingerprint recognition sensor. In various embodiments, the electronic device 400 may recognize the user's iris using an iris recognition sensor. According to various embodiments, the electronic device 400 may automatically perform a Miracast-based service (or function) in response to recognition of the user's biometric information (eg, fingerprint or iris).

In various embodiments, the electronic device 400 may be connected to an external electronic device 500. In various embodiments, the electronic device 400 may be connected to the external electronic device 500 based on wireless communication (e.g., Bluetooth, Bluetooth Low Energy (BLE), WiFi, etc.).

In various embodiments, the electronic device 400 may establish wireless communication with the external electronic device 500 and transmit a device information request to the external electronic device 500. The electronic device 400 responds to a device information request from the external electronic device 500, providing device information related to connection from the external electronic device 500 to the external electronic device 500, such as an operation mode (e.g., landscape mode, Portrait mode) information, resolution information, display type (e.g., portrait, landscape) information, etc.) can be obtained. The electronic device 400 may transmit a screen or content corresponding to the external electronic device based on the acquired device information and display it on the display of the external electronic device 500. In various embodiments, various examples related to the electronic device 400 performing communication connection in conjunction with the external electronic device 500 and performing (providing) Miracast accordingly will be described in detail with reference to the drawings described later. do.

In various embodiments, the external electronic device 500 may be connected to the electronic device 400 through wireless communication and may include various devices including a display. In various embodiments, the external electronic device 500 is wirelessly connected to the electronic device 400, receives data (e.g., screen, content, etc.) displayed on the electronic device 400, and outputs it through the provided display ( display) may include various output devices. For example, the external electronic device 500 may be connected to the electronic device 400 and wirelessly, such as a monitor 510, a tablet PC 520, a laptop PC 530 (or notebook), a television 540, or a refrigerator 550. It can be connected to and can include a variety of devices, including displays. According to various embodiments, the external electronic device 500 may have a similar configuration to the electronic device 400 described above. For example, the external electronic device 500 includes a control circuit (or processor) corresponding to the processor 210 of FIG. 2 and has more configurations than those of the electronic device 201 shown in FIG. 2. , or fewer configurations. In describing various embodiments below, the external electronic device 500 is a refrigerator as an example.

In various embodiments, the external electronic device 500 may be equipped with a display in one area and may include a communication circuit (eg, a communication module) capable of connecting the electronic device 400 and wireless communication. In various embodiments, the external electronic device 500 may receive data transmitted by the electronic device 400 using a communication module and display the received data using a display.

In various embodiments, the external electronic device 500 may connect wireless communication in response to a connection request from the electronic device 400. When wireless communication is connected, the external electronic device 500 may provide device information to the electronic device 400 in response to a request for device information from the electronic device 400.

In various embodiments, various examples related to the external electronic device 500 performing a communication connection in conjunction with the electronic device 400 and operating based on data received accordingly will be described in detail with reference to the drawings described below. explained.

According to various embodiments, the electronic device 400 and the external electronic device 500 may be directly connected through wireless communication. According to various embodiments, the electronic device 400 and the external electronic device 500 may be connected through a commonly connected access point (AP).

According to various embodiments, the electronic device 400 and the external electronic device 500 may exchange messages through a user input back channel (UIBC). According to various embodiments, the electronic device 400 may obtain device information related to the external electronic device 500 through a negotiation procedure when connecting to Miracast. According to various embodiments, the electronic device 400 may check the resolution and display type related to the external electronic device 500 based on device information. According to various embodiments, the electronic device 400 performs Miracast in response to a user request while performing at least one function (or application), and adjusts the resolution of the external electronic device 500 based on the obtained device information. Shared data corresponding to the direction of the display can be generated and transmitted to the external electronic device 500.

As described above, electronic devices according to various embodiments include a wireless communication unit for Miracast connection and data transmission and reception, a display for displaying data, and a processor functionally connected to the wireless communication unit and the display; , the processor obtains device information from an external electronic device connected to Miracast, determines the resolution and display type of the external electronic device based on the device information, and, if the external electronic device is a vertical display, displays a vertical display. Data corresponding to can be generated based on the resolution, and the data can be transmitted to the external electronic device using the wireless communication unit.

According to various embodiments, the device information may be configured to include resolution and display type supported by the external electronic device. According to various embodiments, the device information may include information about the operation mode of the external electronic device, information about the type of the external electronic device, manufacturer information, model name information, or user input back channel (UIBC). ) can be configured to include information.

According to various embodiments, the processor may be configured to determine whether the external electronic device has a vertical display or whether the external electronic device is operating in the vertical direction based on the display type.

According to various embodiments, the processor determines an optimal resolution common to the external electronic device based on the device information and determines a display type of the external electronic device, and when the display type is a vertical display, the processor It can be configured to create a vertical virtual display with a determined resolution, and to generate data corresponding to the resolution and display type of the external electronic device using the virtual display.

According to various embodiments, the processor is configured to rotate the default horizontal virtual display to the vertical direction to correspond to the vertical display of the external electronic device and generate data using the rotated vertical virtual display. can do.

According to various embodiments, when the display type is a horizontal display, the processor may be configured to generate a horizontal virtual display having the determined resolution.

According to various embodiments, the processor receives coordinates according to user input from the external electronic device through a user input back channel (UIBC), determines the type of the external electronic device in response to receiving the coordinates, and If the external electronic device is a first type device based on a vertical display, a second type device recognizes the coordinates without converting the coordinates, and rotates the external electronic device from the default horizontal direction to the vertical direction for use. If so, it can be configured to perform transformation of the coordinates and recognize the transformed coordinates.

According to various embodiments, the processor may be configured to perform coordinate conversion by converting the x-coordinate to y-coordinate and converting the y-coordinate to x-coordinate in the coordinates.

According to various embodiments, the external electronic device detects a user input while connected to the electronic device and Miracast, determines the orientation of the display in response to the detection of the user input, and determines the orientation of the display in the default horizontal direction. If it is used by rotating in the vertical direction, it can be configured to transform the coordinates of the user input and transmit the converted coordinates to the electronic device.

FIG. 5 is a diagram illustrating an example of providing a service using Miracast between electronic devices according to various embodiments of the present invention.

Referring to FIG. 5, in operation 501, the electronic device 400 (e.g., hereinafter referred to as a source device) and the external electronic device 500 (e.g., hereinafter referred to as a sink device) perform a connection procedure. You can start. For example, the source device 400 and the sink device 500 may perform a negotiation procedure for Miracast connection.

In operation 503, the source device 400 may transmit a request message requesting device information related to the sink device 500 in a negotiation procedure. For example, the source device 400 may request resolution information and display type (e.g., vertical display, horizontal display) information supported by the sink device 500. According to various embodiments, the source device 400 may also request information about the operating mode (eg, landscape mode, portrait mode) in which the sink device 500 is operating.

In operation 505, the sink device 500 may transmit a response message containing device information related to the sink device 500 in response to the device information request of the source device 400 in a negotiation procedure. According to various embodiments, device information provided by the sink device 500 may include resolution information and display type information supported by the sink device 500. According to various embodiments, device information includes information about the operating mode of the sink device 500, information about the type of the sink device 500 (e.g., tablet PC, refrigerator, television, etc.), manufacturer information, model name information, It may further include user input back channel (UIBC) information (e.g., version information), etc. According to various embodiments, the source device 400 may determine whether the sink device 500 has a vertical display based on acquired device information (e.g., display type information). For example, in various embodiments, examples of the format of a request message and a response message can be shown as shown in <Table 1> below.

Request type of source device Response type of sink device wfd2-video-formats: wfd2-video-formats: 1920x1080 wfd2_sec_portrait_display: wfd2_sec_portrait_display: rotation 270

For example, the source device 400 includes information requesting the resolution of the sink device 500 (e.g., wfd2-video-formats) and the display type of the sink device 500 (e.g., wfd2_sec_portrait_display) in the request message. It can be transmitted to the sink device 500. The sink device 500 adds the resolution of the sink device 500 (e.g., wfd2-video-formats: 1920x1080) to the response message, and adds the display type (or operation mode (e.g., operating direction of the display) of the sink device 500. )) can be transmitted to the source device 400 including additional information (wfd2_sec_portrait_display: rotation 270). According to various embodiments, the display type may be provided as information rotated based on resolution. For example, while the resolution of the sink device 500 is provided as 1920x1080 based on the horizontal direction, the sink device 500 is vertically oriented, so that the source device 400 recognizes this by providing angle information rotated relative to the horizontal direction ( Example: rotation 270) can be provided. According to one embodiment, the sync device 500 may provide resolution information (e.g., 1920x1080) and angle information at which the display is rotated in the horizontal direction (e.g., rotation 270). The source device 400 may determine whether the display type of the sink device 500 is portrait or landscape based on horizontal resolution information and rotated angle information.

In operation 507, when device information related to the sink device 500 is confirmed, the source device 400 may complete a negotiation procedure with the sink device 500. For example, the source device 400 may transmit a request message to the sink device 500 and receive device information about the sink device 500 from the sink device 500. When the source device 400 obtains resolution information and display type information supported by the sink device 500 from the received device information, it can complete the negotiation procedure.

In operation 509, the source device 400 may transmit data (eg, screen data or playback content) to the sink device 500 connected for the Miracast service. For example, the source device 400 can check resolution information that the sink device 500 can support and determine a resolution that can be commonly supported by each device. Additionally, the source device 400 may determine whether to rotate the image in accordance with the display type of the sink device 500. According to various embodiments, the source device 400 may convert an image corresponding to the determined resolution and display type, and transmit the converted image to the sink device 500. According to various embodiments, the source device 400 may transmit data periodically, continuously, or in real time. According to various embodiments, the sink device 500 may receive data (e.g., screen data or image frame data corresponding to content) transmitted from the source device 400 and display it through a display.

In operation 511, the sink device 500 may transmit a control message corresponding to the user input to the source device 400 while the Miracast connection is established. According to various embodiments, the sink device 500 may transmit a control message according to user input to the source device 400 through a user input back channel (UIBC). Although not shown in FIG. 5, the source device 400 may process user input, control data, and transmit the controlled data to the sink device 500. Input processing operations for user input according to various embodiments will be described in detail with reference to the drawings described later.

FIG. 6 is a flowchart illustrating a method of providing a Miracast service for an electronic device according to various embodiments of the present invention.

As shown in FIG. 6, FIG. 6 may show an example of providing a Miracast service from an electronic device 400 operating as a source device that provides (shares) data.

Referring to FIG. 6, in operation 601, the processor (e.g., the processors 120 and 210 of FIG. 1 or FIG. 2, hereinafter referred to as the processor 210) of the electronic device 400 (e.g., source device) externally The electronic device 500 can be connected to Miracast. According to various embodiments, the processor 210 may connect the external electronic device 500 and Miracast in response to a user input for the Miracast service. According to one embodiment, the processor 210 executes the Miracast service (e.g., in a state of executing an application to display related screen data, or in a state in which the display is turned on (e.g., displaying a home screen), etc. You can receive user input by selecting the Miracast execution icon.

In operation 603, the processor 210 may request device information from the external electronic device 500 in response to the Miracast connection. According to various embodiments, the processor 210 may transmit a request message to the external electronic device 500 requesting a resolution supported by the external electronic device 500 and a display type (or operation mode) of the external electronic device 500. You can.

In operation 605, the processor 210 may receive device information from the external electronic device 500. According to various embodiments, the external electronic device 500 (e.g., a processor of the external electronic device 500) includes device information related to the external electronic device 500 in response to a device information request from the electronic device 400. A response message may be transmitted to the electronic device 400. According to various embodiments, device information may include resolution information and display type information supported by the external electronic device 500. According to various embodiments, device information includes information about the operation mode of the external electronic device 500, information about the type of the external electronic device 500 (e.g., tablet PC, refrigerator, television, etc.), manufacturer information, and model name. It may further include information, or information of a user input back channel (UIBC).

In operation 607, the processor 210 may determine the display type of the external electronic device 500 based on device information. According to various embodiments, the processor 210 may determine whether the external electronic device 500 has a vertical display based on received device information (eg, display type information). According to various embodiments, the processor 210 determines whether the external electronic device 500 is operating in the vertical direction or horizontally based on received device information (e.g., information about the type of the external electronic device 500, operation mode information). It is also possible to determine whether or not it is operating in that direction.

In operation 609, if the processor 210 determines that the external electronic device 500 has a portrait display (or is operating in a portrait orientation), in operation 611, the processor 210 generates data to correspond to the portrait display (or operating in a portrait orientation). can be created. According to various embodiments, the processor 210 stores data (e.g., screen data, playback data) of the electronic device 400 at a resolution supported by the external electronic device 500 and displays the vertical display of the external electronic device 500. It can be created with data corresponding to the type. An operation for generating data according to various embodiments will be described in detail with reference to the drawings described later.

In operation 613, the processor 210 may transmit the generated data to the external electronic device 500. According to various embodiments, the processor 210 may transmit data through streaming. For example, processor 210 may transmit data periodically, continuously, or in real time.

Figures 7 and 8 are diagrams to explain an example of an operation scenario of the Miracast service according to various embodiments of the present invention.

As shown in FIGS. 7 and 8 , FIGS. 7 and 8 may represent an example of a case where the electronic device 400 operates in mirroring mode among the Miracast service (function). In various embodiments, the mirroring mode may include an operation of displaying screen data of the electronic device 400 (eg, a source device) on the external electronic device 500 (eg, a sink device). For example, in the mirroring mode, the screen displayed on the electronic device 400 and the screen displayed on the external electronic device 500 may be the same. 7 and 8 illustrate the sync device 500 (e.g., refrigerator, etc.) based on a vertical display (e.g., a vertical display with a fixed vertical orientation by default) as an example, but the various embodiments are not limited thereto. . For example, according to various embodiments, it can also operate in a display-based sync device 500 (e.g., a tablet PC, etc.) that is used by rotating the display, which is basically horizontal, to the vertical direction.

Referring to FIG. 7, as shown in examples <701> and <707>, an electronic device 400 (hereinafter referred to as a source device) with a resolution of 2560x1440 and operating in a portrait layout; This may indicate a state in which external electronic devices 500 (hereinafter referred to as sink devices) (e.g., refrigerators) with a resolution of 1920x1080 and a vertical display type of 750 are connected to each other through Miracast. According to various embodiments, the source device 400 may determine that the display type of the sink device 500 is a vertical display 750 and the supported resolution is 1920x1080 through a negotiation procedure with the sink device 500. . For example, the sync device 500 may provide resolution information (e.g., 1920x1080) and information on the angle at which the display is rotated in the horizontal direction (e.g., rotation 270). The source device 400 may determine whether the display type of the sink device 500 is portrait or landscape based on horizontal resolution information and rotated angle information.

The source device 400 may generate data (e.g., screen data) to be transmitted to the sink device 500 to correspond to the determined resolution (e.g., 1920x1080). For example, the source device 400 may change the screen data corresponding to its own resolution (2560x1440) to correspond to the resolution (1920x1080) of the sink device 500 to generate shared data.

According to various embodiments, the source device 400 may generate shared data for a vertical display based on the display type of the sink device 500 being a vertical display 750. For example, as shown in example <703>, when generating shared data, the source device 400 uses a virtual display 710 ( Example: You can rotate the canvas to change it to vertical. According to one embodiment, when generating shared data, the source device 400 may create a virtual display 710 with a resolution of 1920x1080 corresponding to the resolution (1920x1080) of the sink device 500. According to various embodiments, the source device 400 may generate vertical screen data in example <701> with a width-to-height ratio (9:16) within the vertical virtual display 710. For example, the source device 400 may use the vertical virtual display 710 to generate vertical screen data displayed on the source device 400 (e.g., portrait layout). According to various embodiments, the source device 400 may generate shared data without rotating screen data, as shown in examples <701> and <703>. The source device 400 may generate shared data using the virtual display 710 corresponding to the resolution supported by the sink device 500.

According to one embodiment, the virtual display 710 may be provided in a landscape format to basically correspond to a landscape orientation display. The source device 400 rotates the default horizontal virtual display 710 to a vertical form (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 (e.g., rotation angle information) of the sink device 500. degrees) can be rotated. In various embodiments, the source device 400 may determine the rotation and/or direction of rotation of the virtual display 710 based on display type information determined in the negotiation procedure. The source device 400 may generate shared data to be displayed on the vertical display 750 using the rotated virtual display 710.

According to various embodiments, the source device 400, as shown in example <705>, sends shared data generated by the virtual display 710 corresponding to the vertical display 750 to the sink device 500. Image frame data can be generated through a buffer 730 (e.g., frame buffer) for transmission to . The source device 400 may transmit image frame data to the sink device 500 through wireless communication.

According to various embodiments, in the shared data generation step using the virtual display 710, the source device 400 uses the virtual display 710 in the horizontal or vertical direction to determine the display type (or Shared data corresponding to the operating direction of the display) and corresponding resolution can be created. Therefore, the source device 400 may not perform an additional resolution change operation to match the resolution of the sink device 500. Therefore, in various embodiments, encoder change according to resolution change may not be performed.

According to various embodiments, the sink device 500 may immediately display image frame data received from the source device 400, as shown in example <707>. For example, as image frame data tailored to correspond to the vertical display 750 is transmitted from the source device 400, the sink device 500 performs any processing on the received image (e.g., rotation and/or Alternatively, the received image frame data can be displayed immediately without scaling (scaling, etc.). For example, data of the same type as data drawn by the source device 400 through the virtual display 710 (e.g., see example <803>) may be displayed through the vertical display 750.

Referring to FIG. 8, as shown in examples <801> and <807>, an electronic device 400 (hereinafter referred to as a source device) with a resolution of 2560x1440 and operating in a landscape layout is provided. , It may indicate a state in which an external electronic device 500 (hereinafter referred to as a sink device) (e.g., a refrigerator) having a resolution of 1920x1080 and a display type of a vertical display 750 is connected to each other through Miracast. According to various embodiments, the source device 400 may determine that the display type of the sink device 500 is a vertical display 750 and the supported resolution is 1920x1080 through a negotiation procedure with the sink device 500. . For example, the sync device 500 may provide resolution information (e.g., 1920x1080) and information on the angle at which the display is rotated in the horizontal direction (e.g., rotation 270). The source device 400 may determine whether the display type of the sink device 500 is portrait or landscape based on the horizontal resolution information and the rotated angle information. The source device 400 is a sink device. Data (e.g., screen data) to be transmitted at 500 can be generated to correspond to the determined resolution (e.g., 1920x1080). For example, the source device 400 may change the screen data corresponding to its own resolution (2560x1440) to correspond to the resolution (1920x1080) of the sink device 500 to generate shared data.

According to various embodiments, the source device 400 may generate shared data for a vertical display based on the display type of the sink device 500 being a vertical display 750. For example, as shown in example <803>, when generating shared data, the source device 400 rotates the virtual display 710 (e.g., canvas) for drawing (or creating) the shared data. It can be changed to vertical format. According to one embodiment, when generating shared data, the source device 400 may create a virtual display 710 with a resolution of 1920x1080 corresponding to the resolution (1920x1080) of the sink device 500. According to various embodiments, the source device 400 may generate horizontal screen data of example <801> with a width-to-height ratio (16:9) within the vertical virtual display 710. For example, the source device 400 may use the vertical virtual display 710 to generate horizontal screen data displayed on the source device 400 (eg, landscape layout). According to various embodiments, the source device 400 may generate shared data without rotating screen data, as shown in examples <801> and <803>. The source device 400 may generate shared data to be displayed on the vertical display 750 using the virtual display 710 corresponding to the resolution supported by the sink device 500.

According to one embodiment, the virtual display 710 may be provided in a landscape format to basically correspond to a landscape orientation display. The source device 400 rotates the default horizontal virtual display 710 to a vertical form (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 (e.g., rotation angle information) of the sink device 500. degrees) can be rotated.

According to various embodiments, the source device 400, as shown in example <805>, sends shared data generated by the virtual display 710 corresponding to the vertical display 750 to the sink device 500. Image frame data can be generated through a buffer 730 (e.g., frame buffer) for transmission to . The source device 400 may transmit image frame data to the sink device 500 through wireless communication. According to various embodiments, in the shared data generation step using the virtual display 710, the source device 400 uses the virtual display 710 in the horizontal or vertical direction to determine the display type (or Shared data corresponding to the operating direction of the display) and corresponding resolution can be created. Therefore, the source device 400 may not perform an additional resolution change operation to match the resolution of the sink device 500. Therefore, in various embodiments, encoder change according to resolution change may not be performed.

According to various embodiments, the sink device 500 may immediately display image frame data received from the source device 400, as shown in example <807>. For example, as image frame data tailored to correspond to the vertical display 750 is transmitted from the source device 400, the sink device 500 performs any processing on the received image (e.g., rotation and/or scaling, etc.). ), the received image frame data can be displayed immediately. For example, data of the same type as data drawn by the source device 400 through the virtual display 710 (e.g., see example <803>) may be displayed through the vertical display 750.

As seen with reference to FIGS. 7 and 8 , according to various embodiments, the source device 400 generates a virtual display 710 in a horizontal or vertical direction corresponding to the display type of the sink device 500, and , the virtual display 710 can be created with a resolution corresponding to the resolution of the sink device 500. Therefore, according to various embodiments, shared data generated in the source device 400 may, for example, be directly displayed on the display in the corresponding direction of the sink device 500 and provide effects, resulting in image deterioration, etc. can be prevented.

Figures 9 and 10 are diagrams to explain another example of an operation scenario of the Miracast service according to various embodiments of the present invention.

As shown in FIGS. 9 and 10, FIGS. 9 and 10 may show an example of operating in presentation mode among the Miracast services (functions). In various embodiments, the presentation mode displays playback data (e.g., data executed (played) by the electronic device 400, such as video data, game data, document data, etc.) of the electronic device 400 (e.g., source device). An operation in which the electronic device 400 provides an interface related to the control of data displayed through the external electronic device 500 or the external electronic device 500 while displaying through the external electronic device 500 (e.g., a sink device). It can be included. For example, in the presentation mode, the screen displayed on the electronic device 400 may be different from the screen displayed on the external electronic device 500. 9 and 10 illustrate the sync device 500 (e.g., refrigerator, etc.) based on a vertical display (e.g., a vertical display with a fixed vertical orientation by default) as an example, but the various embodiments are not limited thereto. . For example, according to various embodiments, it can also operate in a display-based sync device 500 (e.g., a tablet PC, etc.) that is used by rotating the display, which is basically horizontal, to the vertical direction.

Referring to FIG. 9 , as shown in examples <901> and <907>, the electronic device 400 (hereinafter referred to as source device) has a content resolution of 1920x1080 and reproduces the content in a portrait layout; This may indicate a state in which an external electronic device 500 (hereinafter referred to as a sink device) (e.g., a refrigerator) whose display 750 has a resolution of 1920x1080 and a display type of a vertical display 750 is connected to each other via Miracast. According to various embodiments, the source device 400 may determine that the display type of the sink device 500 is a vertical display 750 and the supported resolution is 1920x1080 through a negotiation procedure with the sink device 500. . For example, the sync device 500 may provide resolution information (e.g., 1920x1080) and information on the angle at which the display is rotated in the horizontal direction (e.g., rotation 270). The source device 400 may determine whether the display type of the sink device 500 is portrait or landscape based on horizontal resolution information and rotated angle information.

The source device 400 may generate data (e.g., playback data) to be transmitted to the sink device 500 to correspond to the determined resolution (e.g., 1920x1080). For example, the source device 400 can generate shared data by changing playback data with a specific resolution (1920x1080) to correspond to the resolution (1920x1080) of the sink device 500.

According to various embodiments, the source device 400 may generate shared data for a vertical display based on the display type of the sink device 500 being a vertical display 750. For example, as shown in example <903>, when generating shared data, the source device 400 rotates the virtual display 710 (e.g., canvas) for drawing (or creating) the shared data. It can be changed to vertical format. According to one embodiment, when generating shared data, the source device 400 may create a virtual display 710 with a resolution of 1920x1080 corresponding to the resolution (1920x1080) of the sink device 500. According to various embodiments, the source device 400 may generate vertical playback data of example <901> with a width-to-height ratio (9:16) within the vertical virtual display 710. For example, the source device 400 may use the vertical virtual display 710 to generate vertical playback data displayed on the source device 400 (e.g., portrait layout). According to various embodiments, the source device 400 may generate shared data without rotating playback data, as shown in examples <901> and <903>. The source device 400 may generate shared data using the virtual display 710 corresponding to the resolution supported by the sink device 500.

According to various embodiments, the source device 400 rotates the default horizontal virtual display 710 to a vertical form (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 of the sink device 500. rotation) can be performed. In various embodiments, the source device 400 may determine the rotation and/or direction of rotation of the virtual display 710 based on display type information determined in the negotiation procedure. The source device 400 may generate shared data to be displayed on the vertical display 750 using the rotated virtual display 710.

According to one embodiment, the virtual display 710 may be provided in a landscape format to basically correspond to a landscape orientation display. The source device 400 rotates the default horizontal virtual display 710 to a vertical form (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 (e.g., rotation angle information) of the sink device 500. degrees) can be rotated.

According to various embodiments, the source device 400, as shown in example <905>, sends shared data generated by the virtual display 710 corresponding to the vertical display 750 to the sink device 500. Image frame data can be generated through a buffer 730 (e.g., frame buffer) for transmission to . The source device 400 may transmit image frame data to the sink device 500 through wireless communication.

According to various embodiments, the sink device 500 may immediately display image frame data received from the source device 400, as shown in example <907>. For example, as image frame data tailored to correspond to the vertical display 750 is transmitted from the source device 400, the sink device 500 performs any processing on the received image (e.g., rotation and/or scaling, etc.). ), the received image frame data can be displayed immediately. For example, data of the same type as data drawn by the source device 400 through the virtual display 710 (e.g., see example <903>) may be displayed through the vertical display 750.

According to various embodiments, as shown in example <901>, the source device 400 sequentially or in parallel transmits playback data played in the source device 400 to the sink device 500. , the user interface 900 related to control of playback data may be displayed on the source device 400 according to the presentation mode. The user interface 900 may be implemented in various forms depending on the type of playback data (eg, game data, document data, video data, etc.).

Referring to FIG. 10, as shown in examples <1001> and <1007>, an electronic device 400 (hereinafter referred to as a source device) where the resolution of the content is 1920x1080 and the content is played in a landscape layout. , It may indicate a state in which an external electronic device 500 (hereinafter referred to as a sink device) (e.g., a refrigerator) whose display 750 has a resolution of 1920x1080 and a display type of a vertical display 750 is connected to each other via Miracast. According to various embodiments, the source device 400 may determine that the display type of the sink device 500 is a vertical display 750 and the supported resolution is 1920x1080 through a negotiation procedure with the sink device 500. . For example, the sync device 500 may provide resolution information (e.g., 1920x1080) and information on the angle at which the display is rotated in the horizontal direction (e.g., rotation 270). The source device 400 may determine whether the display type of the sink device 500 is portrait or landscape based on horizontal resolution information and rotated angle information.

The source device 400 may generate data (e.g., playback data) to be transmitted to the sink device 500 to correspond to the determined resolution (e.g., 1920x1080). For example, the source device 400 can generate shared data by changing playback data with a specific resolution (1290x1080) to correspond to the resolution (1920x1080) of the sink device 500.

According to various embodiments, the source device 400 may generate shared data for a vertical display based on the display type of the sink device 500 being a vertical display 750. For example, as shown in example <1003>, when generating shared data, the source device 400 rotates the virtual display 710 (e.g., canvas) for drawing (or creating) the shared data. It can be changed to vertical format. According to one embodiment, when generating shared data, the source device 400 may create a virtual display 710 with a resolution of 1920x1080 corresponding to the resolution (1920x1080) of the sink device 500. According to various embodiments, the source device 400 may generate horizontal screen data in example <1001> with a width-to-height ratio (16:9) within the vertical virtual display 710. For example, the source device 400 may use the vertical virtual display 710 to generate horizontal playback data displayed on the source device 400 (eg, landscape layout). According to various embodiments, the source device 400 may generate shared data without rotating screen data, as shown in examples <1001> and <1003>. The source device 400 may generate shared data to be displayed on the vertical display 750 using the virtual display 710 corresponding to the resolution supported by the sink device 500.

According to one embodiment, the virtual display 710 may be provided in a landscape format to basically correspond to a landscape orientation display. The source device 400 rotates the default horizontal virtual display 710 to a vertical form (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 (e.g., rotation angle information) of the sink device 500. degrees) can be rotated.

According to various embodiments, the source device 400, as shown in example <1005>, sends shared data generated by the virtual display 710 corresponding to the vertical display 750 to the sink device 500. Image frame data can be generated through a buffer 730 (e.g., frame buffer) for transmission to . The source device 400 may transmit image frame data to the sink device 500 through wireless communication.

According to various embodiments, the sink device 500 may immediately display image frame data received from the source device 400, as shown in example <1007>. For example, as image frame data tailored to correspond to the vertical display 750 is transmitted from the source device 400, the sink device 500 performs any processing on the received image (e.g., rotation and/or scaling, etc.). ), the received image frame data can be displayed immediately. For example, data of the same type as data drawn by the source device 400 through the virtual display 710 (e.g., see example <1003>) may be displayed through the vertical display 750.

According to various embodiments, as shown in example <901>, the source device 400 sequentially or in parallel transmits playback data played in the source device 400 to the sink device 500. , the user interface 1000 related to control of playback data may be displayed on the source device 400 according to the presentation mode. The user interface 1000 may be implemented in various forms depending on the type of playback data (eg, game data, document data, video data, etc.).

As seen with reference to FIGS. 9 and 10 , according to various embodiments, the source device 400 generates a virtual display 710 in a horizontal or vertical direction corresponding to the display type of the sink device 500, and , the virtual display 710 can be created with a resolution corresponding to the resolution of the sink device 500. Therefore, according to various embodiments, shared data generated in the source device 400 may, for example, be directly displayed on the display in the corresponding direction of the sink device 500 and provide effects, resulting in image deterioration, etc. can be prevented.

FIG. 11 is a flowchart illustrating an operation method for providing a Miracast service of an electronic device according to various embodiments of the present invention.

As shown in FIG. 11, FIG. 11 may show an example of providing a Miracast service from an electronic device 400 operating as a source device that provides (shares) data.

Referring to FIG. 11, in operation 1101, the processor (e.g., the processors 120 and 210 of FIG. 1 or FIG. 2, hereinafter referred to as the processor 210) of the electronic device 400 (e.g., source device) externally The electronic device 500 can be connected to Miracast. According to various embodiments, the processor 210 may connect the external electronic device 500 and Miracast in response to a user input for the Miracast service.

In operation 1103, the processor 210 may obtain device information related to the external electronic device 500 from the external electronic device 500. For example, device information can be obtained based on operations 603 and 605 of FIG. 6 described above. According to various embodiments, device information may include resolution information and display type information supported by the external electronic device 500. According to various embodiments, device information includes information about the operation mode of the external electronic device 500, information about the type of the external electronic device 500 (e.g., tablet PC, refrigerator, television, etc.), manufacturer information, and model name. It may further include information, or information of a user input back channel (UIBC).

In operation 1105, the processor 210 may analyze the obtained device information. For example, the processor 210 may parse the obtained device information to obtain information about the resolution and display type (or operation mode direction) related to the external electronic device 500.

In operation 1107, the processor 210 may determine an optimal resolution common to the external electronic device 500 and the display type (or operation mode direction) of the external electronic device 500 based on device information.

In operation 1109, the processor 210 may determine whether the display type of the external electronic device 500 is a vertical display based on device information. Alternatively, according to various embodiments, the processor 210 may determine whether the external electronic device 500 is operating in the vertical direction (e.g., the display is rotated to the vertical direction) based on device information.

In operation 1109, if the processor 210 determines that the display type is a vertical display (example of operation 1109), the processor 210 may generate a vertical virtual display with the determined resolution in operation 1111. According to one embodiment, as described in the description with reference to FIGS. 7 to 10, the processor 210 changes the virtual display in the default horizontal form to a vertical form to correspond to the vertical display of the external electronic device 500. You can rotate it (e.g. 90 degrees or 270 degrees). According to various embodiments, the processor 210 operates in operation 1111 even when the external electronic device 500, which can operate in either the horizontal or vertical direction, is operating in the vertical direction (e.g., the display is rotated to the vertical direction). You can create a vertical virtual display as shown.

In operation 1109, if the processor 210 determines that the display type is not a vertical display (No in operation 1109), for example, if it determines that the display type is a horizontal display, in operation 1113, the processor 210 generates a horizontal virtual display with the determined resolution. You can create a display. According to one embodiment, the processor 210 may use a default horizontal virtual display. According to various embodiments, the processor 210 operates in operation 1113 even when the external electronic device 500, which can operate in both the horizontal and vertical directions, is operating in the horizontal direction (e.g., the display is rotated to the horizontal direction). You can create a horizontal virtual display as shown.

In operation 1115, the processor 210 may generate data corresponding to the resolution and display type of the external electronic device 500 using the virtual display generated in operation 1111 or 1113.

In operation 1117, the processor 210 may transmit the generated data to the external electronic device 500 using wireless communication. According to various embodiments, the processor 210 may transmit data through streaming.

FIG. 12 is a diagram illustrating an example of an operation for processing user input during Miracast service according to various embodiments of the present invention.

As shown in FIG. 12, in various embodiments, FIG. 12 may represent an example of an operation for processing a user input (eg, a touch) when a sink device generates it.

Referring to FIG. 12, in operation 1201, the electronic device 400 (hereinafter, a source device) and the external electronic device 500 (hereinafter, a sink device) may be connected to Miracast as described above.

In operation 1203, the sink device 500 may detect a user input. For example, a user may touch an area on the vertical display 750, and the sync device 500 may detect the user's touch.

In operation 1205, when the sink device 500 detects a user input, it may transmit the user input (e.g., a control message) to the source device 400 through a user input back channel (UIBC). According to various embodiments, the sink device 500 controls the source device 400 in response to the user input from the sink device 500 without separate input processing (e.g., coordinate conversion, etc.) for the user input. The touched coordinate value can be transmitted. According to various embodiments, in the case of a vertical display-based sink device 500 with a fixed vertical orientation by default, the width and height each have (x, y), so the coordinates are unchanged as (x, y) without processing. y) Coordinates can be transmitted. According to various embodiments, in the case of the display-based sync device 500 that is used by rotating the display, which is basically horizontal, to the vertical direction, the width and height each have oppositely rotated (y, x), so in the coordinates , x coordinates (e.g. width) can be converted to y coordinates, y coordinates (e.g. height) can be converted to x coordinates, and the converted (x, y) coordinates can be transmitted.

In operation 1207, when a user input is received from the sink device 500, the source device 400 may process the received user input. For example, the source device 400 may process display turn-on or content (or function) related control, etc. based on user input.

In operation 1209, when processing of the user input is completed, the source device 400 may transmit the input processing result to the sink device 500. In various embodiments, FIG. 10 illustrates an example in which the source device 400 transmits an input processing result to the sink device 500, but the various embodiments are not limited thereto. For example, the source device 400 may not transmit the input processing result to the sink device 500. In this case, the user can check whether the user input is processed through a change in the screen of the sink device 500 that is transmitted and output from the source device 400.

Meanwhile, in FIG. 10, the sink device 500 controls the source device 400 by transmitting user input through a user input back channel, but various embodiments are not limited to this. For example, the source device 400 may detect a user input and transmit a control message in response to it to the sink device 500.

FIG. 13 is a flowchart illustrating a method of processing user input in an electronic device according to various embodiments of the present invention.

As shown in FIG. 13 , FIG. 13 may represent an example in which an electronic device 400 operating as a source device providing (sharing) data receives user input and performs related processing.

Referring to FIG. 13, in operation 1301, the processor (e.g., the processors 120 and 210 of FIG. 1 or FIG. 2, hereinafter referred to as the processor 210) of the electronic device 400 (e.g., source device) externally The electronic device 500 (e.g., sink device) may be connected to Miracast. According to various embodiments, the processor 210 may connect the external electronic device 500 and Miracast in response to a user input for the Miracast service.

At operation 1303, processor 210 may detect user input. For example, the processor 210 may detect an internal user input directly generated by the electronic device 400 or an external user input indirectly generated (or received) by the external electronic device 500.

At operation 1305, the processor 210 may determine whether the user input is an external user input. For example, the processor 210 determines whether the user input is a user input received through a user input back channel (UIBC) from the external electronic device 500 connected to Miracast, or a user input generated by the electronic device 400. You can determine whether it is recognized or not.

In operation 1305, if the processor 210 determines that the input is not an external user input (No in operation 1305), the processor 210 may proceed to operation 1311 and process the operations following operation 1311.

If in operation 1305 the processor 210 determines that the input is an external user input (yes in operation 1305), the processor 210 may determine the type of the external electronic device 500 in operation 1307. For example, the processor 210 may determine the type (e.g., display type, operation mode, etc.) of the external electronic device 500 based on device information obtained while performing a negotiation procedure when connecting Miracast. You can.

In operation 1309, based on the result determined in operation 1307, the processor 210 determines whether the external electronic device 500 is a vertical display-based device (e.g., a refrigerator having a vertical display with a default vertical orientation, etc.). ) (hereinafter, a first type device) or a device that can be operated in both the horizontal or vertical directions (e.g., a tablet PC, etc.) (hereinafter, a second type device).

In operation 1309, if the processor 210 determines that the external electronic device 500 is a first type device (example of operation 1309), in operation 1311, data control according to the user input is performed without converting the received user input. It can be handled. For example, in the case of a vertical display with the default vertical orientation, the width and height each have (x, y), so the (x, y) point is recognized as is without processing coordinate conversion. Thus, the corresponding operation can be processed.

If in operation 1309 the processor 210 determines that the external electronic device 500 is a second type device (No in operation 1309), the processor 210 may perform coordinate value conversion for the user input in operation 1313. For example, when a display whose default orientation is horizontal is rotated to portrait, the width and height each have (y, x) rotated in the opposite direction, so in coordinates, the x coordinate (e.g. width) is the y coordinate. , and y-coordinates (e.g. height) can be converted to x-coordinates. For example, the processor 210 may convert (y, x) coordinates to (x, y) coordinates.

In operation 1311, the processor 210 may process data control according to user input based on the converted coordinate value. For example, the processor 210 may recognize the converted (x, y) point and process an operation for it.

In operation 1317, the processor 210 may transmit the data controlled in operation 1311 or 1315 to the external electronic device 500.

According to various embodiments, in FIG. 13, the electronic device 400 determines whether to convert the user input received through the user input back channel (UIBC) according to the type of the external electronic device 500, and determines the result. Although an example of converting the coordinate value of a user input based on the above is shown, various embodiments are not limited to this. For example, when the external electronic device 500 identifies an operation mode and transmits coordinate values converted accordingly, the electronic device 400 may not perform an operation related to converting the received user input. For example, in FIG. 13, operations 1305, 1307, 1309, 1313, and 1315 may not be performed and may be omitted.

FIG. 14 is a flowchart illustrating a method of processing user input in an electronic device according to various embodiments of the present invention.

As shown in FIG. 14 , FIG. 14 may show an example of performing related processing on a user input in an external electronic device 500 operating as a sink device that receives (shares) data. For example, FIG. 14 shows an example of processing a user input in an external electronic device 500 (eg, a tablet PC) that can operate in either the horizontal or vertical direction. According to various embodiments, in FIG. 14, the external electronic device 500 may be an example of a device that operates with its basic operation mode set to the landscape orientation.

Referring to FIG. 14, in operation 1401, the processor (e.g., the processors 120 and 210 of FIG. 1 or FIG. 2, hereinafter referred to as processor 210) of the external electronic device 500 (e.g., sink device) The electronic device 400 (e.g., source device) may be connected to Miracast. According to various embodiments, the processor 210 may connect the electronic device 400 and Miracast in response to a connection request from the electronic device 400.

At operation 1403, processor 210 may detect user input. For example, the processor 210 may detect an internal user input directly generated from the external electronic device 500.

In operation 1405, the processor 210 may determine whether the display is operating in a vertical direction. According to various embodiments, in FIG. 14 it may be assumed that the display of the external electronic device 500 is basically horizontal.

In operation 1405, the processor 210 may determine whether the external electronic device 500 is operating in the vertical direction.

In operation 1405, if the processor 210 determines that the external electronic device 500 is not operating in the portrait orientation (No in operation 1405), for example, the processor 210 determines that the external electronic device 500 operates based on the display in the landscape orientation, which is the default operation mode. In this case, in operation 1407, a control message according to the user input may be transmitted to the electronic device 400 (eg, source device). According to various embodiments, the processor 210 sends a control message (e.g., (x, y) coordinates) corresponding to a user input related to controlling the electronic device 400 through a user input back channel (UIBC) to the electronic device ( 400). According to various embodiments, in the case of a display that is basically horizontal, the width and height have each of (x, y), so the processor 210 can transmit (x, y) coordinates as is without processing to transform the coordinates. there is.

In operation 1405, if the processor 210 determines that the external electronic device 500 is operating in the portrait orientation (example of operation 1405), for example, the processor 210 changes a display whose basic operation mode is landscape to a portrait display format. When used by rotating, in operation 1409, the coordinate value for the user input may be converted. According to various embodiments, when the processor 210 is used by rotating the display, which is basically horizontal, to the vertical direction, the width and height have the oppositely rotated (y, x), respectively, so in the coordinates, the x coordinate ( Example: width) can be converted to y-coordinate, and y-coordinate (eg: height) can be converted to x-coordinate. For example, the processor 210 may convert (y, x) coordinates to (x, y) coordinates.

In operation 1411, the processor 210 may transmit a control message according to the user input to the electronic device 400 (eg, a source device) based on the converted coordinate value. According to various embodiments, the processor 210 sends a control message (e.g., converted (x, y) coordinates) corresponding to a user input related to controlling the electronic device 400 through a user input back channel (UIBC). It can be transmitted to device 400.

In operation 1413, the processor 210 may receive changed data in response to the control message in operation 1407 or operation 1411 and display the received data on a display.

Figures 15 and 16 are diagrams illustrating operation scenarios for processing user input during Miracast service according to various embodiments of the present invention.

As shown in FIGS. 15 and 16, the electronic device 400 (hereinafter, source device) and the external electronic device 500 (hereinafter, sink device) each have a specific resolution (e.g., 1920x1080) and are connected to the Miracaster. It can indicate status. For example, the source device 400 may generate data according to the resolution determined in a negotiation procedure for Miracast connection with the sink device 500 and provide it to the sink device 500.

Referring to FIG. 15, FIG. 15 may show an example of processing user input in a vertical display-based sink device 500 (eg, refrigerator, etc.). According to various embodiments, the sync device 500 in FIG. 15 may be an example of a device having a vertical display with the vertical orientation fixed as a default.

As shown in FIG. 15, when a touch input is detected at a P point (e.g., (x, y) coordinates) while the sink device 500 is connected to the source device 400 by Miracast, the coordinates for the P point are displayed. (e.g. (x, y) coordinates) can be identified. The sink device 500 may transmit the (x, y) coordinates of the sink device 500 to the source device 400 as is without converting the coordinates of the touch input. For example, in the case of a vertical display with the default vertical orientation, the width and height each have (x, y), so the (x, y) coordinates of the sink device 500 are used as the source source without any coordinate conversion processing. It can be delivered to the device 400. According to various embodiments, when the resolution of the source device 400 and the sink device 500 are different, the coordinates (e.g., (x, y) coordinates) for the P point are corrected to match the resolution of the source device 400. Additional actions may be included. This correction may be processed by the source device 400 or the sink device 500, which is determined as the operating entity in a negotiation procedure between devices.

The source device 400 may perform a function corresponding to the coordinates received from the sink device 500 (e.g., (x, y) coordinates of the sink device). The source device 400 may selectively display and transmit data (e.g., screen data, playback data) that changes according to function performance to the sink device 500. For example, in mirroring mode, the source device 400 can transmit while displaying changed data, and in presentation mode, it can transmit without displaying changed data.

Referring to FIG. 16 , FIG. 16 may show an example of processing a user input in a sink device 500 (eg, a tablet PC, etc.) that can operate in either the horizontal or vertical directions. According to various embodiments, in FIG. 16, the sink device 500 may be an example of a device that operates with the basic operation mode set to landscape.

As shown in FIG. 16, when a touch input is detected at a P point (e.g., (y, x) coordinates) while the sink device 500 is connected to the source device 400 by Miracast, the coordinates for the P point are displayed. (e.g. (y, x) coordinates) can be identified. The sink device 500 may convert the coordinates for the touch input and transmit the converted (x, y) coordinates of the sink device to the source device 400. For example, when the sync device 500 uses a display with a default horizontal orientation rotated to the vertical orientation, the width and height have (y, x) each rotated in the opposite direction, so in the coordinates, : Width) can be converted to y coordinates, and y coordinates (e.g. height) can be converted to x coordinates. For example, the sync device 500 may convert the (y, x) coordinates of the touch-input P point into (x, y) coordinates. According to various embodiments, when the resolution of the source device 400 and the sink device 500 are different, the coordinates (e.g., (x, y) coordinates) for the P point are corrected to match the resolution of the source device 400. Additional actions may be included. This correction may be processed by the source device 400 or the sink device 500, which is determined as the operating entity in a negotiation procedure between devices.

The source device 400 may perform a function corresponding to the coordinates received from the sink device 500 (e.g., (x, y) coordinates of the sink device). The source device 400 may selectively display and transmit data (e.g., screen data, playback data) that changes according to function performance to the sink device 500. For example, in mirroring mode, the source device 400 can transmit while displaying changed data, and in presentation mode, it can transmit without displaying changed data. According to various embodiments, when the sink device 500 does not include a coordinate transformation function, coordinate transformation may be processed by the source device 400, as described above in the description with reference to FIG. 13.

As discussed above, the method of operating an electronic device according to various embodiments includes a process of obtaining device information from an external electronic device connected to Miracast, and determining the resolution and display type of the external electronic device based on the device information. A process including, if the external electronic device is a vertical display, generating data corresponding to the vertical display based on the resolution, and transmitting the data to the external electronic device using the wireless communication unit. can do.

According to various embodiments, the device information may include resolution and display type supported by the external electronic device. According to various embodiments, the device information may include information about the operation mode of the external electronic device, information about the type of the external electronic device, manufacturer information, model name information, or user input back channel (UIBC). ) may include information.

According to various embodiments, the determining process may include determining whether the external electronic device is a vertical display or whether the external electronic device is operating in the vertical direction based on the display type. .

According to various embodiments, the generating process includes determining an optimal resolution common to the external electronic device based on the device information and determining a display type of the external electronic device, wherein the display type is a vertical display. In this case, the process may include generating a vertical virtual display having the determined resolution, and using the virtual display to generate data corresponding to the resolution and display type of the external electronic device.

According to various embodiments, the generating process includes rotating a default horizontal virtual display in the vertical direction to correspond to the vertical display of the external electronic device, and data using the rotated vertical virtual display. It may include the process of creating a .

According to various embodiments, the generating process may include generating a horizontal virtual display having the determined resolution when the display type is a horizontal display.

According to various embodiments, a process of receiving coordinates according to user input from the external electronic device through a user input back channel (UIBC), a process of determining the type of the external electronic device in response to receiving the coordinates, and the external electronic device If the electronic device is a first type device based on a vertical display, a process of recognizing the coordinates without converting the coordinates, and if the external electronic device is a second type device that rotates from the default horizontal direction to the vertical direction and is used , It may include the process of performing transformation of the coordinates and recognizing the transformed coordinates.

According to various embodiments, the process of converting the coordinates may include performing coordinate conversion by converting the x-coordinate to a y-coordinate and converting the y-coordinate to an x-coordinate in the coordinates.

According to various embodiments, the external electronic device includes a process of detecting a user input while connected to the electronic device and Miracast, a process of determining the direction of the display in response to the detection of the user input, and a process of determining the direction of the display by default. When used by rotating from the horizontal direction to the vertical direction, the process of converting coordinates for the user input and transmitting the converted coordinates to the electronic device may be further included.

The various embodiments of the present invention disclosed in this specification and drawings are merely provided as specific examples to easily explain the technical content of the present invention and to facilitate understanding of the present invention, and are not intended to limit the scope of the present invention. Therefore, the scope of the present invention should be construed as including all changes or modified forms derived based on the technical idea of the present invention in addition to the embodiments disclosed herein.

400: Electronic device (source device)
120, 210: processor
500: External electronic device (sink device)
710: Virtual display

Claims (20)

In electronic devices,
A wireless communication unit for Miracast connection and data transmission and reception;
a display for displaying data; and
Comprising a processor functionally connected to the wireless communication unit and the display, wherein the processor,
Obtain device information from an external electronic device connected to Miracast,
Determine the resolution and display type of the external electronic device based on the device information,
The display type includes at least one of a first type based on a vertical display or a second type that is used by rotating from the default horizontal direction to the vertical direction,
Generate data corresponding to the display type of the external electronic device based on the resolution,
Using the wireless communication unit, transmit the data to the external electronic device,
Receive coordinates according to user input from the external electronic device through a user input back channel (UIBC),
In response to receiving the coordinates, determine a display type of the external electronic device,
If the display type of the external electronic device is the first type, recognize the coordinates without converting the coordinates,
If the display type of the external electronic device is the second type, perform conversion of the coordinates to recognize the converted coordinates, and
An electronic device configured to control the external electronic device based on the recognized coordinates.
The method of claim 1, wherein the device information is:
An electronic device configured to include a resolution and display type supported by the external electronic device.
The method of claim 2, wherein the device information is:
An electronic device set to include information about the operation mode of the external electronic device, information about the type of the external electronic device, manufacturer information, model name information, or user input back channel (UIBC) information.
The method of claim 2, wherein the processor:
An electronic device configured to determine whether the external electronic device has a vertical display or whether the external electronic device is operating in a vertical direction based on the display type.
The method of claim 1, wherein the processor:
Based on the device information, determine an optimal resolution common to the external electronic device and determine a display type of the external electronic device,
If the display type is a vertical display, create a vertical virtual display with the determined resolution, and
An electronic device configured to generate data corresponding to the resolution and display type of the external electronic device using the virtual display.
The method of claim 5, wherein the processor:
An electronic device configured to rotate a default horizontal virtual display to the vertical direction to correspond to the vertical display of the external electronic device and generate data using the rotated vertical virtual display.
The method of claim 5, wherein the processor:
When the display type is a horizontal display, an electronic device configured to generate a horizontal virtual display with the determined resolution.
delete The method of claim 1, wherein the processor:
In the above coordinates, an electronic device configured to perform coordinate conversion by converting x coordinates to y coordinates and converting y coordinates to x coordinates.
The method of claim 1, wherein the external electronic device:
Detect user input while the electronic device is connected to Miracast,
In response to detecting user input, determine the orientation of the display,
If the orientation of the display is rotated from the default horizontal direction to the vertical direction, the coordinates for the user input are converted, and
An electronic device configured to transmit the converted coordinates to the electronic device.
In a method of operating an electronic device,
The process of obtaining device information from an external electronic device connected to Miracast,
The process of determining the resolution and display type of the external electronic device based on the device information,
The display type includes at least one of a first type based on a vertical display or a second type that is used by rotating from the default horizontal direction to the vertical direction,
A process of generating data corresponding to the display type of the external electronic device based on the resolution,
A process of transmitting the data to the external electronic device using a wireless communication unit,
A process of receiving coordinates according to user input from the external electronic device through a user input back channel (UIBC),
A process of determining a display type of the external electronic device in response to receiving the coordinates,
If the display type of the external electronic device is the first type, recognizing the coordinates without converting the coordinates,
If the display type of the external electronic device is the second type, performing conversion of the coordinates and recognizing the converted coordinates, and
A method including controlling the external electronic device based on the recognized coordinates.
◈Claim 12 was abandoned upon payment of the setup registration fee.◈ The method of claim 11, wherein the device information is:
A method including resolution and display type supported by the external electronic device.
◈Claim 13 was abandoned upon payment of the setup registration fee.◈ The method of claim 12, wherein the device information is:
A method including information about the operation mode of the external electronic device, information about the type of the external electronic device, manufacturer information, model name information, or user input back channel (UIBC) information.
◈Claim 14 was abandoned upon payment of the setup registration fee.◈ The method of claim 12, wherein the determining process is:
A method including determining whether the external electronic device has a vertical display or whether the external electronic device is operating in a vertical direction based on the display type.
◈Claim 15 was abandoned upon payment of the setup registration fee.◈ The method of claim 11, wherein the generating process includes:
A process of determining an optimal resolution common to the external electronic device and a display type of the external electronic device based on the device information;
When the display type is a vertical display, a process of creating a vertical virtual display with the determined resolution, and
A method including generating data corresponding to the resolution and display type of the external electronic device using the virtual display.
◈Claim 16 was abandoned upon payment of the setup registration fee.◈ The method of claim 15, wherein the generating process includes:
A process of rotating a default horizontal virtual display to the vertical direction to correspond to the vertical display of the external electronic device, and
A method comprising generating data using a rotated vertical virtual display.
◈Claim 17 was abandoned upon payment of the setup registration fee.◈ The method of claim 15, wherein the generating process includes:
When the display type is a horizontal display, the method includes generating a horizontal virtual display having the determined resolution.
delete ◈Claim 19 was abandoned upon payment of the setup registration fee.◈ The method of claim 11, wherein the process of converting the coordinates includes:
In the coordinates, a method comprising performing coordinate transformation by converting the x coordinate to a y coordinate and converting the y coordinate to an x coordinate.
◈Claim 20 was abandoned upon payment of the setup registration fee.◈ The method of claim 11, wherein the external electronic device:
A process of detecting user input while connected to the electronic device and Miracast,
A process of determining the direction of the display in response to detecting user input,
If the orientation of the display is rotated from the default horizontal direction to the vertical direction, converting coordinates for the user input, and
The method further includes transmitting the converted coordinates to the electronic device.

Images