KR20180025763A - Method and apparatus for providing miracast - Google Patents

Abstract

According to various embodiments of the present invention, a method and an apparatus for providing a miracast service (function) in an electronic apparatus are disclosed. According to various embodiments of the present invention, the electronic apparatus comprises: a wireless communication unit for connecting miracast, and transmitting and receiving data; a display for displaying the data; and a processor for being functionally connected to the wireless communication unit and the display, and configured to obtain apparatus information from an external electronic apparatus connected to the miracast, to determine a display type and resolution of the external electronic apparatus based on the apparatus information, to generate data corresponding to a vertical display based on the resolution when the external electronic apparatus is the vertical display, and to transmit the data to the external electronic apparatus by using the wireless communication unit. Various embodiments are possible.

Description

METHOD AND APPARATUS FOR PROVIDING MIRACAST < RTI ID = 0.0 >

Various embodiments of the present invention disclose methods and apparatus for providing a miracast service (function) in an electronic device.

BACKGROUND ART Recently, with the development of digital technology, a mobile communication terminal, a smart phone, a tablet personal computer, an electronic notebook, a notebook, a personal digital assistant (PDA), a wearable device Various types of electronic devices are widely used.

The electronic device is implemented in the form of a multimedia player having complex functions. For example, the electronic device can be used for communication functions such as voice call and video call, message sending and receiving functions such as SMS (short message service) / MMS (multimedia message service) and e-mail (e-mail) (Eg, HDMI, high definition multimedia interface) communication function, shooting function, broadcast playback function, video playback function, music playback function, internet function, messenger function, game Function, or a social networking service (SNS) function.

In recent years, there has been a demand for services (or functions) using technologies (e.g., miracast) that wirelessly connect (connect) an electronic device to an external electronic device (e.g., a displayable device such as a television, a monitor, ) Have been increasing. For example, a user can display and share a screen displayed on an electronic device through a display of an external electronic device (e.g., a television, a monitor, etc.) to which the electronic device is wirelessly connected. These services (or functions) are provided under the names of miracast, screen mirroring, airplay or WiDi (wireless display) depending on the manufacturer of the electronic device.

On the other hand, the service (or function) by the current miracle cast is specialized when the external electronic device (or display) is in the landscape orientation. Therefore, in the current mirror cast, it is not supported for portrait orientation display, or data of the electronic device can be provided rotated. For example, in a mirror cast, data transmitted from an electronic device based on the resolution (e.g., 1280x1024, 1920x1080, 2560x1440, etc.) of the lateral direction of the external electronic device may be displayed on the display. Therefore, when the external electronic device is in the form of a portrait display, the data can be provided rotated in relation to the resolution in the horizontal direction. For example, the data of the portrait layout of the electronic device is rotated 90 degrees (or -90 degrees) through the display in the longitudinal direction, or the data of the landscape layout of the electronic device is displayed 90 degrees (or -90 degrees) to be provided in the transverse direction. In order to compensate for this, it is conventionally necessary to perform a correction (for example, image processing) in the external electronic device (for example, a sink device) And has the same layout as the layout displayed on the apparatus. However, this may be possible if the external electronic device necessarily supports the function for image processing.

In various embodiments, shared data (e.g., screen data, playback data) corresponding to a portrait display of an external electronic device (e.g., sink device) in an electronic device (e.g., a source device) A method and apparatus capable of transmitting data are disclosed.

In various embodiments, corresponding to the display type of the external electronic device in the electronic device (e.g., a vertical display in which the longitudinal direction is fixed on a basis of a base, a display in which the base is used by rotating the display in the vertical direction) And a method and an apparatus capable of generating a signal.

In various embodiments, when transferring shared data from an electronic device to an external electronic device, shared data corresponding to the display type of the external electronic device is generated, without further transmission of specific information (e.g., operating mode information or content direction information) A method and apparatus capable of transmitting data are disclosed.

In various embodiments, a control message related to control of the electronic device is generated based on the recognized coordinates, without performing different processing on the user input back channel (UIBC) according to the operation mode in the external electronic device A method and an apparatus capable of transmitting data to and from the apparatus.

An electronic device according to various embodiments of the present invention includes a wireless communication unit for a miracast connection and data transmission and reception, a display for data display, and a processor functionally coupled to the wireless communication unit and the display, Determines the resolution and display type of the external electronic device based on the device information, and if the external electronic device is a vertical display, Generating data based on the resolution, and using the wireless communication unit to transmit the data to the external electronic device.

A method of operating an electronic device according to various embodiments of the present invention includes: acquiring device information from a mirror-connected external electronic device; determining a resolution and a display type of the external electronic device based on the device information; Generating the data corresponding to the vertical display based on the resolution when the external electronic device is a vertical display and transmitting the data to the external electronic device using the wireless communication unit .

In order to solve the above problems, various embodiments of the present invention may include a computer-readable recording medium recording a program for causing the processor to execute the method.

According to the electronic device and its operating method according to various embodiments, shared data (e.g., screen data, playback data, etc.) corresponding to a longitudinal display of an external electronic device (e.g., a sink device) ) Can be generated and transmitted to the external electronic device. For example, according to various embodiments, a display type (e.g., a vertical display in which the longitudinal direction is fixed to the base, a display in which the base is used to rotate the display in the vertical direction in the vertical direction) The shared data can be generated. Thus, the external electronic device can immediately display the received shared data without further processing (e.g., shared data scaling, shared data rotation, etc.) on the received shared data. According to various embodiments, since the electronic device provides shared data in accordance with the resolution and display type of the external electronic device, there is no additional processing on the shared data in the external electronic device, so that image deterioration due to scaling or rotation can be reduced .

According to various embodiments, the electronic device generates shared data corresponding to the display type of the external electronic device, without further transmission of specific information (e.g., operation mode information or content direction information) when transmitting the shared data to the external electronic device . Therefore, the electronic device may not perform processing (for example, encoder change, etc.) in accordance with the resolution change or the like, thereby reducing current consumption. According to various embodiments, processing of the user input back channel (UIBC) according to the operation mode of the external electronic device is processed in the same manner, so that user input can be recognized and processed more quickly. The electronic device according to various embodiments can contribute to improving usability, convenience, and competitiveness of the electronic device.

1 is a diagram illustrating a network environment including an electronic device according to various embodiments of the present invention.
2 is a block diagram of an electronic device in accordance with various embodiments of the present invention.
3 is a block diagram of a program module in accordance with various embodiments of the present invention.
4 is a diagram illustrating an example of electronic devices according to various embodiments of the present invention.
5 is a diagram illustrating an example of providing a service using miracast among electronic devices according to various embodiments of the present invention.
Figure 6 is a flow diagram illustrating an operational method for providing a mirror cast service of an electronic device in accordance with various embodiments of the present invention.
7 and 8 are diagrams illustrating an example of an operation scenario of a mirror cast service according to various embodiments of the present invention.
9 and 10 are diagrams illustrating another example of an operation scenario of a miracast service according to various embodiments of the present invention.
11 is a flow chart illustrating an operational method of providing a mirror cast service of an electronic device in accordance with various embodiments of the present invention.
12 is a diagram illustrating an exemplary operation for processing user input in a mirror cast service in accordance with various embodiments of the present invention.
13 is a flow diagram illustrating an operational method for processing user input in an electronic device in accordance with various embodiments of the present invention.
14 is a flow chart illustrating an operational method for processing user input in an electronic device in accordance with various embodiments of the present invention.
15 and 16 are diagrams illustrating operation scenarios for handling user input in a mirror cast service in accordance with various embodiments of the present invention.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. It is to be understood that the embodiments and terminologies used herein are not intended to limit the invention to the particular embodiments described, but to include various modifications, equivalents, and / or alternatives of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar components. The singular expressions may include plural expressions unless the context clearly dictates otherwise. In this document, the expressions "A or B" or "at least one of A and / or B" and the like may include all possible combinations of the items listed together. Expressions such as "first," " second, "" first, " or "second, " But are not limited to these components. When it is mentioned that some (e.g., first) component is "(functionally or communicatively) connected" or "connected" to another (second) component, May be connected directly to the component, or may be connected through another component (e.g., a third component).

In this document, the term "configured to" as used herein is intended to encompass all types of information, including, but not limited to, "hardware" or "software" Quot ;, " modified to ", "made to be "," capable of ", or "designed to" interchangeably. In some situations, the expression "a device configured to" may mean that the device can "do " with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be implemented by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) And a generic-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) capable of performing the corresponding operations.

Electronic devices in accordance with various embodiments of the present document may be used in various applications such as, for example, smart phones, tablet PCs, mobile phones, videophones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, personal digital assistants an assistant, a portable multimedia player (PMP), an MP3 player, a medical device, a camera, or a wearable device. Wearable devices may be of the accessory type (e.g., watches, rings, bracelets, braces, necklaces, glasses, contact lenses or head-mounted-devices), textile or garment- (E. G., A skin pad or tattoo), or a bio-implantable circuit. In some embodiments, the electronic device may include at least one of a television, a DVD (eg, Samsung HomeSync ^™ , Apple TV), digital video disc player, audio, refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air purifier, set top box, home automation control panel, ^TM , or Google TV ^TM ), a game console (e.g., Xbox ^TM , PlayStation ^TM ), an electronic dictionary, an electronic key, a camcorder, or an electronic frame.

In another embodiment, the electronic device may be used in a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a temperature meter), magnetic resonance angiography (MRA) A navigation system, a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), an automobile infotainment device, a ship electronics Equipment such as marine navigation systems, gyro compass, avionics, security devices, head units for vehicles, industrial or home robots, drone, automated teller machines (ATMs) , Point of sale (POS) of shops, or internet items of things (eg light bulbs, sensors, sprinkler devices, fire alarms, thermostats, street lights, toasters, exercise equipment, hot water tanks,It may comprise at least one of a boiler, etc.). According to some embodiments, the electronic device may be a piece of furniture, a building / structure or part of an automobile, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (e.g., Gas, or radio wave measuring instruments, etc.). In various embodiments, the electronic device is flexible or may be a combination of two or more of the various devices described above. The electronic device according to the embodiment of the present document is not limited to the above-described devices. In this document, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

1 is a diagram illustrating a network environment including an electronic device according to various embodiments.

Referring to Figure 1, in various embodiments, an electronic device 101 in a network environment 100 is described. The electronic device 101 may include a bus 110, a processor 120, a memory 130, an input / output interface 150, a display 160, and a communication interface 170. In some embodiments, the electronic device 101 may omit at least one of the components or additionally comprise other components.

The bus 110 may include circuitry to connect the components 110-170 to one another and to communicate communications (e.g., control messages or data) between the components.

Processor 120 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 may perform computations or data processing related to, for example, control and / or communication of at least one other component of the electronic device 101. The processing (or control) operation of the processor 120 according to various embodiments is specifically described with reference to the following figures.

The memory 130 may include volatile memory and / or non-volatile memory. Memory 130 may store instructions or data related to at least one other component of electronic device 101, for example. According to one embodiment, the memory 130 may store software and / or programs 140. The program 140 may include, for example, a kernel 141, a middleware 143, an application programming interface (API) 145, and / or an application program, (Or "application") 147 and the like. At least a portion of the kernel 141, middleware 143, or API 145 may be referred to as an operating system (OS).

The memory 130 may store one or more programs executed by the processor 120 and may perform functions for temporary storage of input / output data. According to various embodiments, the memory 130 is responsible for storing the acquired data, the data acquired in real time may be stored in a temporary storage device (e.g., a buffer), and the data determined to be stored may be stored It can be stored in a storage device. The memory 130 may include a computer-readable recording medium having recorded thereon a program for causing the processor 120 to execute a method according to various embodiments.

The kernel 141 may include system resources used to execute an operation or function implemented in other programs (e.g., middleware 143, API 145, or application program 147) (E.g., bus 110, processor 120, or memory 130). The kernel 141 also provides an interface to control or manage system resources by accessing individual components of the electronic device 101 in the middleware 143, API 145, or application program 147 .

The middleware 143 can perform an intermediary role such that the API 145 or the application program 147 can communicate with the kernel 141 to exchange data. In addition, the middleware 143 may process one or more task requests received from the application program 147 according to the priority order. For example, middleware 143 may use system resources (e.g., bus 110, processor 120, or memory 130, etc.) of electronic device 101 in at least one of application programs 147 Prioritize, and process the one or more task requests. The API 145 is an interface for the application 147 to control functions provided by the kernel 141 or the middleware 143 such as file control, window control, At least one interface or function (e.g., command) for character control and the like.

The input / output interface 150 may be configured to communicate commands or data entered from a user or an external device to another component (s) of the electronic device 101, ) To the user or an external device. For example, a wired / wireless headphone port, an external charger port, a wired / wireless data port, a memory card port, an audio input / output port, a video input / And the like may be included in the input / output interface 150.

The display 160 may be a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, an active OLED ), Micro-electromechanical systems (MEMS) displays, or electronic paper displays, and the like. Display 160 may display various content (e.g., text, images, video, icons, and / or symbols, etc.) to a user, for example. Display 160 may include a touchscreen and may receive touch, gesture, proximity, or hovering input, for example, using an electronic pen or a portion of the user's body.

Display 160 may, for example, show the user a visual output. The visual output may appear in the form of text, graphics, video, and combinations thereof. The display 160 can display (output) various information processed in the electronic device 101. [ For example, the display 160 may display a user interface (UI) or a graphical user interface (GUI) associated with use of the electronic device.

In accordance with various embodiments, the display 160 may display various UIs or GUIs associated with operations performed by the electronic device 101. In the electronic device 101 according to various embodiments, various screen examples provided on the basis of the user interface will be described with reference to the following drawings.

The communication interface 170 establishes communication between the electronic device 101 and an external device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106) . For example, communication interface 170 may be connected to network 162 via wireless or wired communication to communicate with an external device (e.g., second external electronic device 104 or server 106).

Wireless communication may be, for example, long term evolution (LTE), LTE Advance, code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) , Or global system for mobile communications (GSM), and the like. According to one embodiment, the wireless communication may include, for example, wireless fidelity, wireless gigabit alliance, Bluetooth, Bluetooth low energy, Zigbee, NFC field communication, a magnetic secure transmission, a radio frequency (RF), or a body area network (BAN). According to one embodiment, the wireless communication may comprise a GNSS. The GNSS may be, for example, a global positioning system (GPS), a global navigation satellite system (Glonass), a Beidou Navigation Satellite System (Beidou) or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, " GPS " can be used interchangeably with " GNSS ".

The wired communication may include, for example, a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 232 (RS-232), a power line communication, or a plain old telephone service And may include at least one.

The network 162 may include at least one of a telecommunications network, e.g., a computer network (e.g., a local area network (LAN) or wide area network (WAN) .

Each of the first external electronic device 102 and the second external electronic device 104 may be the same or a different kind of device as the electronic device 101. [ According to various embodiments, all or a portion of the operations performed in the electronic device 101 may be performed in another electronic device or multiple electronic devices (e.g., electronic device 102, 104, or server 106). According to one embodiment, in the event that the electronic device 101 has to perform certain functions or services automatically or on demand, the electronic device 101 may be capable of executing the function or service itself, (E. G., Electronic device 102, 104, or server 106) at least some of the associated functionality. Other electronic devices (e. G., Electronic device 102, 104, or server 106) may execute the requested function or additional function and deliver the result to electronic device 101. [ The electronic device 101 can directly or additionally process the received result to provide the requested function or service. For this purpose, for example, cloud computing, distributed computing, or client-server computing techniques may be used.

2 is a block diagram of an electronic device according to various embodiments.

The electronic device 201 may include all or part of the electronic device 101 shown in Fig. 1, for example. The electronic device 201 may include one or more processors (e.g., AP) 210, a communication module 220, a subscriber identification module 224, a memory 230, a sensor module 240, an input device 250, a display 260, an interface 270, an audio module 280, a camera module 291, a power management module 295, a battery 296, an indicator 297, and a motor 298. In various embodiments, the electronic device 201 may be implemented with more configurations than the configurations shown in FIG. 2, or with fewer configurations, as the configurations shown in FIG. 2 are not essential. For example, the electronic device 201 according to various embodiments may not include any of some components depending on its type. According to various embodiments, the configurations of the above-described electronic device 201 may be seated on the housing (or bezel, body) of the electronic device 201, or formed on the outside thereof.

The processor 210 may, for example, drive an operating system or application program to control a number of hardware or software components coupled to the processor 210, and may perform various data processing and operations. The processor 210 may be implemented with, for example, a system on chip (SoC). According to one embodiment, the processor 210 may further include a graphics processing unit (GPU) and / or an image signal processor (ISP). Processor 210 may include at least some of the components shown in FIG. 2 (e.g., cellular module 221). Processor 210 may load and process instructions or data received from at least one of the other components (e.g., non-volatile memory) into volatile memory and store the resulting data in non-volatile memory.

In various embodiments, the processor 210 may control the overall operation of the electronic device 201. In various embodiments, the processor 210 may include one or more processors. For example, processor 210 may include a communications processor (CP), an application processor (AP), an interface (e.g., general purpose input / output (GPIO) Can be integrated into an integrated circuit. According to one embodiment, the application processor may execute various software programs to perform various functions for the electronic device 201, and the communication processor may perform processing and control for voice communication and data communication Can be performed. The processor 210 may execute a specific software module (e.g., an instruction set) stored in the memory 230 to perform various specific functions corresponding to the module.

Processor 210 may control the operation of hardware modules such as audio module 280, interface 270, display 260, camera module 291, communication module 220, and the like. According to various embodiments, the processor 210 may be electrically coupled to the display 260 and the memory 230 of the electronic device 201.

According to various embodiments, the processor 210 may perform operations to obtain device information from a mirrored external electronic device 500, determine the resolution and display type of the external electronic device 500 based on the device information, If the electronic device 500 is a vertical display, it can handle operations to generate data corresponding to the vertical display based on the resolution, and to transmit data to the external electronic device 500 using the wireless communication unit . According to various embodiments, the processor 210 determines the optimal resolution common with the external electronic device 500 based on the device information, and determines the display type of the external electronic device 500, and determines whether the display type is a vertical display A vertical virtual display having a determined resolution, and, if the display type is a horizontal display, processing operations related to generating a horizontal virtual display having a determined resolution. In accordance with various embodiments, the processor 210 may utilize the virtual display to handle operations related to generating data corresponding to the resolution and display type of the external electronic device 500.

The processing (or control) operation of the processor 210 in accordance with various embodiments is specifically described with reference to the following figures.

The communication module 220 may have the same or similar configuration as, for example, the communication interface 170 shown in Fig. The communication module 220 may include, for example, a cellular module 221, a WiFi module 223, a Bluetooth module 225, a GNSS module 227, an NFC module 228 and an RF module 229 have. Although not shown, the communication module 220 may further include, for example, a WiGig module (not shown). According to one embodiment, the WiFi module 223 and the WiGig module (not shown) may be integrated into one chip.

The cellular module 221 may provide, for example, voice calls, video calls, text services, or Internet services over a communication network. According to one embodiment, the cellular module 221 may perform identification and authentication of the electronic device 201 within the communication network using a subscriber identification module (e.g., SIM (subscriber identification module) card 224) . According to one embodiment, the cellular module 221 may perform at least some of the functions that the processor 210 may provide. According to one embodiment, the cellular module 221 may comprise a communications processor (CP). At least some (e.g., two or more) of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228, according to some embodiments, integrated chip) or an IC package.

The RF module 229 can, for example, send and receive communication signals (e.g., RF signals). The RF module 229 may include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228 transmits / receives an RF signal through a separate RF module .

The WiFi module 223 may represent a module for forming a wireless LAN link with, for example, a wireless Internet connection and an external device (e.g., another electronic device 102 or a server 106, etc.). The WiFi module 223 may be embedded or enclosed in the electronic device 201. WiFi, WiGig, WiBro, world interoperability for microwave access (WiMax), high speed downlink packet access (HSDPA), or mmWave (millimeter wave) can be used as the wireless Internet technology. The WiFi module 223 may be connected to the electronic device via a direct connection (e.g., peer to peer connection) or an external connection (e.g., via a network 162) In association with a device (e.g., another electronic device 104), various data of the electronic device 201 can be transmitted to the outside or can be received from the outside. The WiFi module 223 may be kept on at all times or may be turned on / off according to the setting of the electronic device or user input.

The Bluetooth module 225 and the NFC module 228 may represent, for example, a short range communication module for performing short range communication. Bluetooth, low power Bluetooth (BLE), radio frequency identification (RFID), infrared communication (IrDA), ultra wideband (UWB), Zigbee, or NFC may be used as a short distance communication technology. The short-range communication module is configured to interoperate with an external device (e.g., another electronic device 102 or the like) connected to the electronic device 201 via a network (e.g., a local area communication network) The device can be transmitted or received. The short range communication module (e.g., Bluetooth module 225 and NFC module 228) may be kept on all the time or may be turned on / off according to the setting of the electronic device 201 or the user input.

The subscriber identification module 224 may include, for example, a card or an embedded SIM containing a subscriber identity module, and may include unique identification information (e.g., ICCID) or subscriber information (e.g., IMSI (international mobile subscriber identity).

Memory 230 (e.g., memory 130) may include, for example, internal memory 232 or external memory 234. The built-in memory 232 may be implemented as, for example, volatile memory (e.g., dynamic random access memory (DRAM), synchronous random access memory (SRAM), or synchronous dynamic random access memory (SDRAM) (ROM), a flash ROM, a flash memory, a hard drive, or a solid state drive (SSD) such as a single time programmable read only memory (ROM), a programmable ROM (PROM), an erasable and programmable ROM (EPROM) and a solid state drive). The external memory 234 may be a flash drive, for example, a compact flash (CF), a secure digital (SD), a micro-SD, a Mini- An SD, an extreme digital (xD), a multi-media card (MMC), a memory stick, etc. The external memory 234 may be functionally or physically connected to the electronic device 201 through various interfaces.

The memory 230 may be used by the processor 210 to determine the resolution and display type of the external electronic device 500 based on the device information, One associated with generating the data corresponding to the vertical display based on the resolution and transmitting data to the external electronic device 500 using the wireless communication unit if the external electronic device 500 is a vertical display Or more programs, data, or instructions. According to various embodiments, the memory 230 determines the optimal resolution resolution common to the external electronic device 500 and the display type of the external electronic device 500 based on the device information, and if the display type is a vertical display Data, or instructions associated with generating a vertical virtual display having a determined resolution and generating a horizontal virtual display having a determined resolution if the display type is a horizontal display have. According to various embodiments, the memory 230 may store one or more programs, data, or instructions associated with generating data corresponding to the resolution and display type of the external electronic device 500 using the virtual display .

The memory 230 may include an extended memory (e.g., external memory 234) or an internal memory (e.g., internal memory 232). The electronic device 201 may operate in association with web storage that performs the storage function of the memory 230 over the Internet.

The memory 230 may store one or more software (or software modules). For example, a software component may be an operating system software module, a communications software module, a graphics software module, a user interface software module, a moving picture experts group (MPEG) module, a camera software module, . ≪ / RTI > A module, which is a software component, can also be expressed as a set of instructions, so a module is sometimes referred to as an instruction set. Modules can also be expressed as programs. In various embodiments of the present invention, the memory 230 may include additional modules (instructions) in addition to the modules described above. Or may not use some modules (commands) as needed.

Operating system software modules may include various software components that control general system operations. Control of this general system operation may mean, for example, memory management and control, storage hardware (device) control and management, or power control and management. In addition, an operating system software module can also facilitate the communication between various hardware (devices) and software components (modules).

The communication software module may enable communication with other electronic devices, such as a wearable device, a smart phone, a computer, a server, a refrigerator, a television, a monitor or a portable terminal, via the communication module 220 or interface 270. The communication software module may be configured with a protocol structure corresponding to the communication method.

The graphics software module may include various software components for providing and displaying graphics on the display 260. In various embodiments, the term graphic may be used to mean including text, a web page, an icon, a digital image, video, animation, etc. have.

The user interface software module may include various software components related to the user interface (UI). For example, how the state of the user interface is changed or under which conditions the change of the user interface state is made, and the like.

An MPEG module may include software components that enable processes and functions related to digital content (e.g., video, audio), such as creation, playback, distribution, and transmission of content.

The camera software module may include camera-related software components that enable camera-related processes and functions.

An application module may include a web browser, a email, an instant message, a word processing, a keyboard emulation, an address book, etc., including a rendering engine. Touch list, widget, digital right management (DRM), iris scan, context cognition, voice recognition, positioning (position) determining function, location based service, and the like. According to various embodiments of the present invention, the application module includes instructions for displaying and sharing the screen (or content) displayed on the electronic device 201 through the display of an external electronic device to which the electronic device 210 is wirelessly connected can do.

The sensor module 240 may, for example, measure a physical quantity or sense the operating state of the electronic device 201 to convert the measured or sensed information into an electrical signal. The sensor module 240 includes a gesture sensor 240A, a gyro sensor 240B, a barometer sensor 240C, a magnetic sensor 240D, An acceleration sensor 240E, a grip sensor 240F, a proximity sensor 240G, a color sensor 240H (e.g., RGB (red, green, blue) At least one of a temperature sensor 240J, a temperature-humidity sensor 240J, an illuminance sensor 240K, or a UV (ultra violet) sensor 240M. One can be included. Additionally or alternatively, the sensor module 240 may include sensors such as, for example, an e-nose sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor An electrocardiogram (ECG) sensor, an infrared (IR) sensor, an iris scan sensor, and / or a finger scan sensor. The sensor module 240 may further include a control circuit for controlling at least one or more sensors belonging to the sensor module 240. In some embodiments, the electronic device 201 further includes a processor configured to control the sensor module 240, either as part of the processor 210 or separately, so that while the processor 210 is in a sleep state, The sensor module 240 can be controlled.

The input device 250 may include, for example, a touch panel 252, a (digital) pen sensor 254, a key 256, or an ultrasonic input device 258. As the touch panel 252, for example, at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type can be used. Further, the touch panel 252 may further include a control circuit. The touch panel 252 may further include a tactile layer to provide a tactile response to the user. (Digital) pen sensor 254 may be part of, for example, a touch panel or may include a separate recognition sheet. Key 256 may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device 258 can sense the ultrasonic wave generated from the input tool through the microphone 288 and confirm the data corresponding to the ultrasonic wave detected. According to various embodiments, the input device 250 may include an electronic pen. According to various embodiments, the input device 250 may be implemented to receive a force touch.

Display 260 (e.g., display 160) may include panel 262, hologram device 264, projector 266, and / or control circuitry for controlling them.

The panel 262 may be embodied, for example, flexibly, transparently, or wearably. The panel 262 may comprise a touch panel 252 and one or more modules. According to one embodiment, the panel 262 may include a pressure sensor (or force sensor) capable of measuring the intensity of the pressure on the user's touch. The pressure sensor may be integrated with the touch panel 252 or may be implemented by one or more sensors separate from the touch panel 252. The panel 262 may be seated in the display 260 and may sense user input contacting or approaching the surface of the display 260. The user input may comprise a touch input or proximity input based on at least one of a single-touch, a multi-touch, a hovering, or an air gesture. The panel 262 may receive user input to initiate an operation associated with use of the electronic device 201 in various embodiments and may generate an input signal in accordance with the user input. The panel 262 may be configured to convert a change in capacitance, such as a pressure applied to a particular area of the display 260 or a specific area of the display 260, to an electrical input signal. The panel 262 can detect the location and area where an input tool (e.g., a user finger, an electronic pen, etc.) is touched or approximated on the surface of the display 260. In addition, the panel 262 can be configured to detect pressure (e.g., force touch) at the time of touch according to the applied touch method.

The hologram device 264 can display a stereoscopic image in the air using interference of light. The projector 266 can display an image by projecting light onto a screen. The screen may be located, for example, inside or outside the electronic device 201.

The interface 270 may include, for example, an HDMI 272, a USB 274, an optical interface 276, or a D-sub (D-subminiature) 278. The interface 270 may, for example, be included in the communication interface 170 shown in FIG. Additionally or alternatively, the interface 270 may include, for example, a mobile high-definition link (MHL) interface, an SD card / multi-media card (MMC) interface, or an infrared data association have.

The interface 270 may receive data from another electronic device, or may receive power and communicate it to the respective configurations within the electronic device 201. [ The interface 270 may allow data within the electronic device 201 to be transmitted to other electronic devices. For example, a wired / wireless headphone port, an external charger port, a wired / wireless data port, a memory card port, an audio input / output port, a video input / Etc. may be included in the interface 270.

The audio module 280 can, for example, convert sound and electrical signals in both directions. At least some of the components of the audio module 280 may be included, for example, in the input / output interface 145 shown in FIG. The audio module 280 may process sound information input or output through, for example, a speaker 282, a receiver 284, an earphone 286, a microphone 288, or the like. The audio module 280 transmits the audio signal input from the processor 210 to an output device (e.g., a speaker 282, a receiver 284, or an earphone 286) And transmits an audio signal, such as a voice, received from the processor 210 to the processor 210. The audio module 280 converts audio / sound data into audible sound through an output device under the control of the processor 210, and converts the audio signal, such as voice, received from the input device into a digital signal, Lt; / RTI >

The speaker 282 or the receiver 284 may receive audio data from the communication module 220 or stored in the memory 230. [ The speaker 282 or the receiver 284 may output an acoustic signal associated with various operations (functions) performed in the electronic device. The microphone 288 can receive an external acoustic signal and process it as electrical voice data. The microphone 288 may be implemented with various noise reduction algorithms for eliminating noise generated in receiving an external sound signal. The microphone 288 may be responsible for input of audio streaming such as voice commands or the like.

The camera module 291 is, for example, a device capable of capturing still images and moving images, and according to one embodiment, one or more image sensors (e.g., a front sensor or a rear sensor), a lens, an image signal processor (ISP) , Or flash (e.g., an LED or xenon lamp, etc.).

According to various embodiments, the camera module 291 may include, for example, a first camera (e.g., a color camera) to acquire color information and depth information (e.g., (E.g., an infrared (IR) camera) for acquiring information (e.g., information). For example, a first camera (e.g., a color camera) can convert a light input from the outside into a video signal and photograph a color image of the subject. The camera module 291 may include an image sensor. The image sensor may be implemented as a CCD (charged coupled device) or a CMOS (complementary metal-oxide semiconductor). According to one embodiment, the first camera may be a front camera provided on the front surface of the electronic device 201. [ In accordance with various embodiments, the front camera may be replaced by a second camera, and may not be provided at the front of the electronic device 201. According to various embodiments, the first camera may be placed together with the second camera on the front of the electronic device 201 together. According to one embodiment, the first camera may be a rear camera provided on the rear surface of the electronic device 201. [ According to one embodiment, the first camera may be configured to include both a front camera and a rear camera, which are provided on the front and back sides of the electronic device 201, respectively.

The power management module 295 can, for example, manage the power of the electronic device 201. [ According to one embodiment, the power management module 295 may include a power management integrated circuit (PMIC), a charger IC, a battery, or a fuel gauge. The PMIC may have a wired and / or wireless charging scheme. The wireless charging scheme may include, for example, a magnetic resonance scheme, a magnetic induction scheme, or an electromagnetic wave scheme, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonant circuit, have. The battery gauge can measure, for example, the remaining amount of the battery 296, the voltage during charging, the current, or the temperature. The battery 296 may include, for example, a rechargeable battery and / or a solar battery.

The indicator 297 may indicate a particular state of the electronic device 201 or a portion thereof (e.g., processor 210), e.g., a boot state, a message state, or a state of charge. The motor 298 can convert the electrical signal to mechanical vibration, and can generate vibration, haptic effects, and the like. Electronic device 201 is, for example, DMB Mobile TV-enabled devices capable of handling media data in accordance with standards such as (digital multimedia broadcasting), DVB (digital video broadcasting), or MediaFLO (mediaFlo ^TM) (for example, : GPU).

Each of the components described in this document may be composed of one or more components, and the name of the component may be changed according to the type of the electronic device. In various embodiments, an electronic device (e.g., electronic device 101, 201) may have some components omitted, additional components included, or some of the components being combined into one entity , The functions of the corresponding components before the combination can be performed in the same manner.

3 is a block diagram of a program module in accordance with various embodiments.

According to one embodiment, program module 310 (e.g., program 140) includes an operating system that controls resources associated with an electronic device (e.g., electronic device 101, 201) and / Application (e.g., application program 147). The operating system may include, for example, Android ^TM , iOS ^TM , Windows ^TM , Symbian ^TM , Tizen ^TM , or Bada ^TM .

3, the program module 310 includes a kernel 320 (e.g., a kernel 141), a middleware 330 (e.g., middleware 143), an API 360 (e.g., API 145) , And / or an application 370 (e.g., an application program 147). At least a portion of the program module 310 may be preloaded on the electronic device or may be downloaded from an external electronic device such as the electronic device 102 104 or the server 106,

The kernel 320 may include, for example, a system resource manager 321 and / or a device driver 323. The system resource manager 321 can perform control, allocation, or recovery of system resources. According to one embodiment, the system resource manager 321 may include a process manager, a memory manager, or a file system manager. The device driver 323 may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WiFi driver, an audio driver, or an inter-process communication . The middleware 330 may provide various functions through the API 360, for example, to provide functions that are commonly needed by the application 370 or allow the application 370 to use limited system resources within the electronic device. Application 370 as shown in FIG.

According to one embodiment, middleware 330 includes a runtime library 335, an application manager 341, a window manager 342, a multimedia manager 343, A resource manager 344, a power manager 345, a database manager 346, a package manager 347, a connectivity manager 348, A notification manager 349, a location manager 350, a graphic manager 351, or a security manager 352, as shown in FIG. have.

The runtime library 335 may include, for example, a library module that the compiler uses to add new functionality via a programming language while the application 370 is executing. The runtime library 335 may perform input / output management, memory management, or arithmetic function processing.

The application manager 341 can manage the life cycle of the application 370, for example. The window manager 342 can manage graphical user interface (GUI) resources used in the screen. The multimedia manager 343 can recognize the format required for reproducing the media files and can perform encoding or decoding of the media file using a codec according to the format. The resource manager 344 can manage the source code of the application 370 or the space of the memory. The power manager 345 may, for example, manage the capacity or power of the battery and provide the power information necessary for operation of the electronic device. According to one embodiment, the power manager 345 may interoperate with a BIOS (basic input / output system). The database manager 346 may create, retrieve, or modify the database to be used in the application 370, for example. The package manager 347 can manage installation or update of an application distributed in the form of a package file.

The connectivity manager 348 may, for example, manage the wireless connection. The notification manager 349 may provide the user with an event such as, for example, an arrival message, an appointment, a proximity notification, and the like. The location manager 350 can manage the location information of the electronic device, for example. The graphic manager 351 can manage, for example, a graphic effect to be provided to the user or a user interface related thereto. The security manager 352 may provide, for example, system security or user authentication.

According to one embodiment, the middleware 330 may include a telephony manager for managing the voice or video call function of the electronic device, or a middleware module capable of forming a combination of the functions of the above-described components. According to one embodiment, the middleware 330 may provide a module specialized for each type of operating system. Middleware 330 may dynamically delete some existing components or add new ones.

The API 360 may be provided in a different configuration depending on the operating system, for example, as a set of API programming functions. For example, for Android or iOS, you can provide a single API set for each platform, and for Tizen, you can provide two or more API sets for each platform.

The application 370 may include a home 371, a dialer 372, an SMS / MMS 373, an instant message 374, a browser 375, a camera 376, an alarm 377, Contact 378, voice dial 379, email 380, calendar 381, media player 382, album 383, watch 384, healthcare (e.g., measuring exercise or blood glucose) , Or environmental information (e.g., air pressure, humidity, or temperature information) application.

According to one embodiment, the application 370 may include an information exchange application capable of supporting the exchange of information between the electronic device and the external electronic device. The information exchange application may include, for example, a notification relay application for communicating specific information to an external electronic device, or a device management application for managing an external electronic device. For example, the notification relay application may transmit notification information generated in another application of the electronic device to the external electronic device, or receive notification information from the external electronic device and provide the notification information to the user. The device management application may, for example, turn-on / turn-off the function of an external electronic device in communication with the electronic device (e.g., the external electronic device itself ) Or adjusting the brightness (or resolution) of the display), or installing, deleting, or updating an application running on an external electronic device. According to one embodiment, the application 370 may include an application (e.g., a healthcare application of a mobile medical device) designated according to the attributes of the external electronic device.

According to one embodiment, the application 370 may include an application received from an external electronic device. At least some of the program modules 310 may be implemented (e.g., executed) in software, firmware, hardware (e.g., processor 210), or a combination of at least two of the same, Program, routine, instruction set or process.

As used herein, the term "module" includes a unit of hardware, software or firmware and may be, for example, a logic, a logic block, a component, ) And the like can be used interchangeably. A "module" may be an integrally constructed component or a minimum unit or part thereof that performs one or more functions. "Module" may be implemented either mechanically or electronically, for example, by application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs) And may include programmable logic devices. At least a portion of a device (e.g., modules or functions thereof) or a method (e.g., operations) according to various embodiments includes instructions stored in a computer readable storage medium (e.g., memory 130) . ≪ / RTI > When the instruction is executed by a processor (e.g., processor 120), the processor may perform a function corresponding to the instruction.

The computer readable recording medium may be a hard disk, a floppy disk, magnetic media (e.g., magnetic tape), optical recording media (e.g., compact disc read only memory (CD-ROM) a digital versatile disc, magneto-optical media (e.g., a floptical disk), internal memory, etc. The instructions may be executed by code or an interpreter generated by the compiler The module or program module according to various embodiments may include at least one or more of the elements described above, some of which may be omitted, or may further include other elements. Operations performed by a module, program module or other component, according to the examples, may be performed sequentially, in parallel, repetitively, or heuristically, May be executed in a different order, omitted, or other actions may be added.

According to various embodiments, the recording medium may include a computer-readable recording medium having recorded thereon a program for causing the processor 120, 210 to execute the various methods described below.

In various embodiments, the electronic device may include any device that uses one or more of a variety of processors, such as an AP, a CP, a GPU, and a CPU. For example, an electronic device according to various embodiments may include an information communication device, a multimedia device, a wearable device, an IoT device, or various other devices corresponding to those devices.

Hereinafter, an operation method and apparatus according to various embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted, however, that the various embodiments of the present invention are not limited or limited by the following description, and can be applied to various embodiments based on the following embodiments. In the various embodiments of the present invention described below, a hardware approach will be described as an example. However, various embodiments of the present invention include techniques using both hardware and software, so that various embodiments of the present invention do not exclude a software-based approach.

4 is a diagram illustrating an example of electronic devices according to various embodiments of the present invention.

4, a system in accordance with various embodiments of the present invention includes an electronic device 400 (e.g., a first electronic device or source device) and at least one external electronic device 500 : A second electronic device or sink device).

In various embodiments, the electronic device 400 includes a display 417, a housing (or body) 450 to which the display 417 is seated and fastened, An additional device for performing the operation, and the like. In various embodiments, the additional device may include a first speaker 401, a second speaker 403, a microphone 405, a sensor (e.g., a front camera 407, an illuminance sensor 409, etc.), a communication interface A data input / output port 411, an audio input / output port 413, etc.), a button 415, and the like.

In various embodiments, the display 417 may include a curved display (or a bended display) that can be bent, bent, or otherwise undamaged through a thin, flexible substrate, such as a planar display or paper . The curved display can be coupled to the housing 450 (or a bezel to maintain a curved shape). In various embodiments, the electronic device 400 is bent like a flexible display, such as a curved display. In various embodiments, the display 417 may be a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), or an active OLED In various embodiments, the display 417 may be mounted on at least one side of the electronic device 400 (e.g., a front side of the electronic device 400), or alternatively may be replaced with a plastic film to replace the glass substrate wrapping the liquid crystal in an AMOLED, active matrix OLED (e.g., at least one of the left, right, upper, and lower sides) of the curved display so that the radius of curvature (e.g., curvature radius of 5 cm, 1 cm, 7.5 mm, 5 mm, 4 mm, and the like) or folded to the side of the housing 450.

In various embodiments, the electronic device 400 can recognize a fingerprint of a user using a fingerprint recognition sensor. In various embodiments, the electronic device 400 can recognize the iris of the user using the iris recognition sensor. According to various embodiments, the electronic device 400 may automatically perform a miracast-based service (or function) in response to recognition of the user's biometric information (e.g. fingerprint or iris).

In various embodiments, the electronic device 400 may be coupled to the external electronic device 500. In various embodiments, the electronic device 400 may be coupled to an external electronic device 500 based on wireless communications (e.g., Bluetooth, low power Bluetooth (BLE), WiFi, etc.).

In various embodiments, the electronic device 400 may connect wireless communication with the external electronic device 500 and send a device information request to the external electronic device 500. The electronic device 400 may respond to the device information request to the external electronic device 500 by sending device information related to the connection from the external electronic device 500 to the external electronic device 500, Vertical mode) information, resolution information, type of display (e.g., vertical and horizontal), and the like). The electronic device 400 can transmit a screen or contents corresponding to the external electronic device based on the acquired device information, and display it through the display of the external electronic device 500. Various examples relating to the electronic device 400 in various embodiments in conjunction with the external electronic device 500 to perform the communication connection and thus perform (provide) the miracast will be described in detail with reference to the drawings do.

In various embodiments, the external electronic device 500 may be in wireless communication with the electronic device 400 and may include various devices including a display. In various embodiments, the external electronic device 500 is wirelessly coupled to the electronic device 400 to receive data (e.g., screen, content, etc.) displayed on the electronic device 400 and output Lt; RTI ID = 0.0 > and / or < / RTI > For example, the external electronic device 500 may be coupled to the electronic device 400, such as a monitor 510, a tablet PC 520, a laptop PC 530 (or a notebook), a television 540 or a refrigerator 550, And may include various devices including a display. According to various embodiments, the external electronic device 500 may have a configuration similar to the electronic device 400 described above. For example, the external electronic device 500 includes control circuitry (or a processor) corresponding to the processor 210 of FIG. 2 and has more configurations than the configurations of the electronic device 201 shown in FIG. 2 , ≪ / RTI > or fewer configurations. In the following description of various embodiments, the external electronic device 500 is a refrigerator.

In various embodiments, the external electronic device 500 may include a communication circuit (e.g., a communication module) that can be mounted with a display in one area and can couple wireless communication with the electronic device 400. In various embodiments, the external electronic device 500 may utilize a communication module to receive data transmitted by the electronic device 400 and to display the received data using a display.

In various embodiments, the external electronic device 500 may connect the wireless communication in response to a connection request of the electronic device 400. The external electronic device 500 may provide the device information to the electronic device 400 in response to the device information request of the electronic device 400 when the wireless communication is connected.

In various embodiments, various examples related to the external electronic device 500 performing a communication connection in conjunction with the electronic device 400 and thus operating based on the received data will be described in detail with reference to the following drawings .

According to various embodiments, the electronic device 400 and the external electronic device 500 may be connected directly to the wireless communication. According to various embodiments, the electronic device 400 and the external electronic device 500 may be connected through an access point (AP) that is connected in common.

According to various embodiments, the electronic device 400 and the external electronic device 500 may exchange messages via a user input back channel (UIBC). According to various embodiments, the electronic device 400 may obtain device information related to the external electronic device 500 through a negotiation procedure when the mirrored connection with the external electronic device 500 is performed. According to various embodiments, the electronic device 400 can determine the resolution, display type, etc. associated with the external electronic device 500 based on the device information. According to various embodiments, the electronic device 400 performs a miracast in response to a user request during at least one function (or application), and performs a mirror cast on the resolution of the external electronic device 500 based on the acquired device information Shared data corresponding to the direction of the display may be generated and transmitted to the external electronic device 500.

As described above, an electronic device according to various embodiments includes a wireless communication unit for a miracast connection and data transmission / reception, a display for data display, and a processor functionally coupled to the wireless communication unit and the display Wherein the processor is configured to: obtain device information from a mirrored external electronic device; determine a resolution and a display type of the external electronic device based on the device information; and if the external electronic device is a vertical display, Based on the resolution, and to transmit the data to the external electronic device using the wireless communication unit.

According to various embodiments, the device information may be configured to include a resolution and a display type supported by the external electronic device. According to various embodiments, the device information includes information about an operating mode of the external electronic device, information on the type of the external electronic device, manufacturer information, model name information, or user input back channel (UIBC) And the like.

According to various embodiments, the processor may be configured to determine, based on the display type, whether the external electronic device is a portrait display or whether the external electronic device is operating in a portrait orientation.

According to various embodiments, the processor may determine an optimal resolution common to the external electronic device based on the device information and a display type of the external electronic device, and if the display type is a vertical display, Generating a vertical virtual display having a determined resolution, and using the virtual display to generate data corresponding to the resolution and display type of the external electronic device.

According to various embodiments, the processor is configured to rotate the virtual display in the default landscape orientation to correspond to the portrait display of the external electronic device, and to generate data using the rotated portrait orientation virtual display can do.

According to various embodiments, the processor may be configured to generate a landscape virtual display having the determined resolution if the display type is a landscape display.

According to various embodiments, the processor is further configured to receive coordinates from a user input via a user input back channel (UIBC) from the external electronic device, determine a type of the external electronic device in response to the coordinate reception, Wherein if the external electronic device is a first type device based on a vertical display, the coordinates are recognized without conversion to the coordinates, and the second type device , It is possible to perform the transformation of the coordinates and to recognize the converted coordinates.

According to various embodiments, the processor may be configured to perform coordinate transformation at the coordinates, wherein the x coordinate is converted to the y coordinate and the y coordinate is converted to the x coordinate.

According to various embodiments, the external electronic device senses user input in a mirror-cast relationship with the electronic device, determines the orientation of the display in response to user input sensed, And to transmit the converted coordinates to the electronic device if the user is in a state of being rotated in the longitudinal direction at the user input.

5 is a diagram illustrating an example of providing a service using miracast among electronic devices according to various embodiments of the present invention.

5, at operation 501, an electronic device 400 (e.g., a source device) and an external electronic device 500 (e.g., sink device) You can start. For example, the source apparatus 400 and the sink apparatus 500 may perform a negotiation procedure for a miracast connection.

At operation 503, the source device 400 may send a request message requesting device information related to the sink device 500 in the negotiation procedure. For example, the source device 400 may request resolution information supported by the sink device 500 and the type of display (e.g., vertical display, horizontal display) information. According to various embodiments, the source device 400 may also request information about the operating mode (e.g., landscape mode, portrait mode) in which the sink device 500 is operating.

In operation 505, the sink device 500 may transmit a response message including the device information related to the sink device 500 in response to the device information request of the source device 400 in the negotiation procedure. According to various embodiments, the device information provided by the sink device 500 may include resolution information and display type information that the sink device 500 supports. According to various embodiments, the device information includes information on the operation mode of the sink device 500, information on the type of the sink device 500 (e.g., tablet PC, refrigerator, television, etc.), manufacturer information, model name information, Information (e.g., version information) of a user input back channel (UIBC), and the like. According to various embodiments, the source device 400 may determine whether the sink device 500 is a vertical display based on the acquired device information (e.g., type information of the display). For example, in various embodiments, a format example of a request message and a response message can be illustrated as shown in Table 1 below.

The request type of the source device The response type of the sink device wfd2-video-formats: wfd2-video-formats: 1920x1080 wfd2_sec_portrait_display: wfd2_sec_portrait_display: rotation 270

For example, the source device 400 may include information requesting the resolution (e.g., wfd2-video-formats) of the sink device 500 and the display type (e.g., wfd2_sec_portrait_display) of the sink device 500 in the request message To the sink device (500). The sink device 500 adds the resolution of the sink device 500 to the response message (for example, wfd2-video-formats: 1920x1080) and sets the display type (or the operation mode of the sink device 500 ) To the source device 400 (wfd2_sec_portrait_display: rotation 270). According to various embodiments, the display type may be provided with rotated information based on the resolution. For example, the resolution of the sink device 500 is 1920 x 1080 in the horizontal direction, while the sink device 500 is in the vertical direction, so that angle information rotated in the horizontal direction to recognize it in the source device 400 For example, rotation 270). According to one embodiment, the sink device 500 may provide resolution information (e.g., 1920x1080) and angle information (e.g., rotation 270) in which the display is rotated in the horizontal direction. The source apparatus 400 can determine whether the display type of the sink device 500 is the vertical direction or the horizontal direction based on the horizontal resolution information and the rotated angle information.

In operation 507, the source device 400 can complete the negotiation procedure with the sink device 500 when the device information related to the sink device 500 is confirmed. For example, the source apparatus 400 may send a request message to the sink apparatus 500 and receive the apparatus information of the sink apparatus 500 from the sink apparatus 500. [ The source apparatus 400 can complete the negotiation procedure when the resolution information and display type information supported by the sink apparatus 500 are obtained from the received apparatus information.

At operation 509, the source device 400 may transmit data (e.g., screen data or playback content) to the sink device 500 connected for the mirrorcast service. For example, the source device 400 can determine the resolutions that can be commonly supported by each device by checking the resolution information that the sink device 500 can support. In addition, the source apparatus 400 may determine whether the image is rotated in correspondence with the display type of the sink apparatus 500. According to various embodiments, the source device 400 may convert the image corresponding to the determined resolution and display type, and may transmit the converted image to the sink device 500. [ According to various embodiments, the source device 400 may transmit data periodically, continuously or in real time. According to various embodiments, sink device 500 may receive data (e.g., image data corresponding to screen data or content) transmitted from source device 400 and display it via a display.

At operation 511, the sink device 500 may transmit a control message corresponding to the user input to the source device 400, with the miracast connection established. In accordance with various embodiments, the sink device 500 may transmit a control message according to user input to the source device 400 via a user input back channel (UIBC). Although not shown in FIG. 5, the source device 400 may process the user input to control the data and transmit the controlled data to the sink device 500. The input processing operation for user input according to various embodiments will be described in detail with reference to the drawings described later.

Figure 6 is a flow diagram illustrating an operational method for providing a mirror cast service of an electronic device in accordance with various embodiments of the present invention.

As shown in FIG. 6, FIG. 6 may show an example of providing a miracast service in an electronic device 400 that acts as a source device for providing (sharing) data.

6, processor (e.g., processors 120 and 210 of FIG. 1 or 2, hereinafter referred to as processor 210) of electronic device 400 (e.g., source device) The electronic device 500 and the mirror cast can be connected. According to various embodiments, the processor 210 may couple the miracast with the external electronic device 500 in response to a user input for the miracast service. According to one embodiment, the processor 210 executes a miracast service (e.g., in a state in which an application is executed to display relevant screen data, or in a state in which the display is turned on (e.g., displaying a home screen) And selecting a mirror cast execution icon).

At operation 603, the processor 210 may request device information from the external electronic device 500 in response to the mirror cast connection. According to various embodiments, the processor 210 may send a request message to the external electronic device 500 requesting the resolution supported by the external electronic device 500 and the display type (or mode of operation) of the external electronic device 500 .

At operation 605, the processor 210 may receive device information from the external electronic device 500. In accordance with various embodiments, an external electronic device 500 (e.g., a processor of an external electronic device 500) may include device information associated with an external electronic device 500 in response to a device information request of the electronic device 400 Response message to the electronic device 400. [ According to various embodiments, the device information may include resolution information supported by the external electronic device 500 and type information of the display. According to various embodiments, the device information includes information about the mode of operation of the external electronic device 500, information about the type of external electronic device 500 (e.g., tablet PC, refrigerator, television, etc.) Information, or information of a user input back channel (UIBC), and the like.

At operation 607, the processor 210 may determine the display type of the external electronic device 500 based on the device information. According to various embodiments, the processor 210 may determine whether the external electronic device 500 is a vertical display based on the received device information (e.g., type information of the display). According to various embodiments, the processor 210 determines whether the external electronic device 500 is operating in the longitudinal direction based on received device information (e.g., information about the type of external electronic device 500, operating mode information) It is possible to judge whether or not it is in operation.

In operation 609, if the processor 210 determines that the external electronic device 500 is a vertical display (or vertically in motion), then in operation 611, the data 210 Lt; / RTI > According to various embodiments, the processor 210 may convert data (e.g., screen data, playback data) of the electronic device 400 to a resolution supported by the external electronic device 500, Can be generated with data corresponding to the type. The operation of generating data in accordance with various embodiments will be described in detail with reference to the drawings described later.

At operation 613, the processor 210 may send the generated data to the external electronic device 500. According to various embodiments, the processor 210 may transmit data streaming when transmitting. For example, the processor 210 may transmit data periodically, continuously, or in real time.

7 and 8 are diagrams illustrating an example of an operation scenario of a mirror cast service according to various embodiments of the present invention.

As shown in Figs. 7 and 8, Figs. 7 and 8 can show an example in which the electronic device 400 operates in a mirroring mode among miracle-cast services (functions). In various embodiments, the mirroring mode may include the act of displaying the screen data of the electronic device 400 (e.g., source device) as it is in the external electronic device 500 (e.g., a sink device). For example, the mirroring mode may be the same as the screen displayed on the electronic device 400 and the screen displayed on the external electronic device 500. 7 and 8 illustrate an example of a sink device 500 (e.g., a refrigerator or the like) based on a vertical display (e.g., a vertical display with a fixed vertical orientation), but the various embodiments are not limited thereto . For example, according to various embodiments, the display may operate in a display-based sink device 500 (e.g., a tablet PC, etc.) that is used to rotate the display in the landscape orientation in the primary direction.

7, there is shown an electronic device 400 (hereinafter referred to as a source device) that operates at a portrait layout with a resolution of 2560x1440, as shown in FIG. 7 and FIG. 7, An external electronic device 500 (hereinafter referred to as a sink device) (for example, a refrigerator) having a resolution of 1920 x 1080 and a display type of which is a vertical display 750 may be mirror-cast to each other. According to various embodiments, the source device 400 may determine through the negotiation procedure with the sink device 500 that the display type of the sink device 500 is the vertical display 750 and the supported resolution is 1920x1080 . For example, the sink device 500 may provide resolution information (e.g., 1920x1080) and angle information (e.g., rotation 270) in which the display is rotated in the horizontal direction. The source apparatus 400 can determine whether the display type of the sink device 500 is the vertical direction or the horizontal direction based on the horizontal resolution information and the rotated angle information.

The source apparatus 400 may generate data (e.g., screen data) to be transmitted to the sink apparatus 500 so as to correspond to the determined resolution (e.g., 1920x1080). For example, the source apparatus 400 can generate the shared data by changing the screen data corresponding to the resolution (2560x1440) of the source apparatus 400 to correspond to the resolution (1920x1080) of the sink apparatus 500. [

According to various embodiments, the source device 400 may generate shared data for a vertical display, based on the display type of the sink device 500 being a vertical display 750. [ For example, as shown in the example, the source device 400 may include a virtual display 710 (FIG. 7B) for drawing (or creating) shared data when generating shared data For example: a canvas can be rotated to change its shape vertically. According to one embodiment, the source device 400 may generate a virtual display 710 with a resolution of 1920x1080 corresponding to the resolution (1920x1080) of the sink device 500 when generating the shared data. According to various embodiments, the source apparatus 400 may generate the screen data of the vertical form of the example in a horizontal and vertical ratio (9:16) within the virtual display 710 of the vertical form. For example, the source apparatus 400 can be generated in the form of vertical screen data (e.g., portrait layout) displayed on the source apparatus 400 using the virtual display 710 in the form of a vertical form. According to various embodiments, the source device 400 may generate shared data without the rotation of the screen data, for example, as shown in FIG. The source device 400 may generate the shared data using the virtual display 710 corresponding to the supported resolution of the sink device 500. [

According to one embodiment, the virtual display 710 may be provided in a landscape format to correspond to a display of essentially landscape orientation. The source device 400 rotates vertically (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 (e.g., rotation angle information) of the sink device 500, Can also be rotated). In various embodiments, the source device 400 may determine the rotation and / or the direction of rotation of the virtual display 710 based on the type information of the display determined in the negotiation procedure. The source device 400 may use the rotated virtual display 710 to generate shared data to be displayed on the vertical display 750.

In accordance with various embodiments, the source apparatus 400 may provide the shared data generated by the virtual display 710, corresponding to the vertical display 750, to the sink device 500, as shown in the example, (E.g., a frame buffer) for transmitting the image frame data to the buffer 730. [ The source device 400 may transmit the image frame data to the sink device 500 via wireless communication.

According to various embodiments, the source device 400 may generate the shared data using the virtual display 710 by using the virtual display 710 in the horizontal or vertical direction, The direction of operation of the display) and the shared data corresponding to that resolution. Therefore, the source apparatus 400 may not perform an additional resolution changing operation to adjust the resolution of the sink apparatus 500. [ Thus, in various embodiments, it may not perform encoder changes due to resolution changes.

In accordance with various embodiments, the sink device 500 may immediately display image frame data received from the source device 400, as shown in the example. For example, as image frame data aligned in the source device 400 to correspond to the vertical display 750 is transmitted, the sink device 500 may perform some processing (e.g., rotation and / Or scaling, etc.) of the received image frame data. For example, data of the same type as the data (e.g., example 803) drawn by the source device 400 via the virtual display 710 may be displayed via the vertical display 750.

8, there is shown an electronic device 400 (hereinafter, referred to as a source device) that operates at a landscape layout with a resolution of 2560x1440, as shown in FIG. 8 and FIG. , An external electronic device 500 (hereinafter referred to as a sink device) (e.g., a refrigerator) having a resolution of 1920 x 1080 and a display type of which is a vertical display 750 may be mirror-cast to each other. According to various embodiments, the source device 400 may determine through the negotiation procedure with the sink device 500 that the display type of the sink device 500 is the vertical display 750 and the supported resolution is 1920x1080 . For example, the sink device 500 may provide resolution information (e.g., 1920x1080) and angle information (e.g., rotation 270) in which the display is rotated in the horizontal direction. The source device 400 can determine whether the display type of the sink device 500 is the vertical direction or the horizontal direction based on the horizontal resolution information and the rotated angle information. (E.g., screen data) to be transmitted to the display unit 500 corresponding to the determined resolution (e.g., 1920x1080). For example, the source apparatus 400 can generate the shared data by changing the screen data corresponding to the resolution (2560x1440) of the source apparatus 400 to correspond to the resolution (1920x1080) of the sink apparatus 500. [

According to various embodiments, the source device 400 may generate shared data for a vertical display, based on the display type of the sink device 500 being a vertical display 750. [ For example, as shown in the example, the source device 400 may rotate a virtual display 710 (e.g., a canvas) to draw (or create) shared data when generating shared data It can be changed into a vertical form. According to one embodiment, the source device 400 may generate a virtual display 710 with a resolution of 1920x1080 corresponding to the resolution (1920x1080) of the sink device 500 when generating the shared data. According to various embodiments, the source device 400 may generate screen data of the landscape of the example in a vertical and vertical aspect ratio (16: 9) within the virtual display 710. For example, the source device 400 can be generated in the form of landscape data (e.g., landscape layout) displayed on the source device 400 using the virtual display 710 in the vertical form. According to various embodiments, the source device 400 may generate shared data without rotation of the screen data, such as example < 801 > and example. The source device 400 may generate the shared data to be displayed on the vertical display 750 using the virtual display 710 corresponding to the supported resolution of the sink device 500. [

According to one embodiment, the virtual display 710 may be provided in a landscape format to correspond to a display of essentially landscape orientation. The source device 400 rotates vertically (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 (e.g., rotation angle information) of the sink device 500, Can also be rotated).

In accordance with various embodiments, the source apparatus 400 may send the shared data generated by the virtual display 710 corresponding to the vertical display 750 to the sink device 500, as shown in the example, (E.g., a frame buffer) for transmitting the image frame data to the buffer 730. [ The source device 400 may transmit the image frame data to the sink device 500 via wireless communication. According to various embodiments, the source device 400 may generate the shared data using the virtual display 710 by using the virtual display 710 in the horizontal or vertical direction, The direction of operation of the display) and the shared data corresponding to that resolution. Therefore, the source apparatus 400 may not perform an additional resolution changing operation to adjust the resolution of the sink apparatus 500. [ Thus, in various embodiments, it may not perform encoder changes due to resolution changes.

According to various embodiments, the sink device 500 may immediately display the image frame data received from the source device 400, as shown in the example. For example, as image frame data aligned in the source device 400 to correspond to the vertical display 750 is transmitted, the sink device 500 may perform some processing (e.g., rotation and / or scaling, etc.) on the received image The received image frame data can be displayed immediately. For example, data of the same type as the data (e.g., example 803) drawn by the source device 400 via the virtual display 710 may be displayed via the vertical display 750.

7 and 8, according to various embodiments, the source apparatus 400 generates a virtual display 710 in the horizontal or vertical direction corresponding to the display type of the sink apparatus 500 , The virtual display 710 can be generated with a resolution corresponding to the resolution of the sink device 500. Thus, according to various embodiments, the shared data generated in the source device 400 can provide, for example, direct display and effects on the corresponding directional display of the sink device 500, Can be prevented.

9 and 10 are diagrams illustrating another example of an operation scenario of a miracast service according to various embodiments of the present invention.

As shown in Figs. 9 and 10, Figs. 9 and 10 illustrate examples of operations in a presentation mode among miracle-cast services (functions). In various embodiments, the presentation mode may be configured to store playback data (e.g., data played (played) by electronic device 400, such as video data, game data, or document data) of electronic device 400 The operation of providing the interface associated with the control of the displayed data via the external electronic device 500 or the external electronic device 500 while displaying it via the external electronic device 500 (e.g., a sink device) . For example, in the presentation mode, the screen displayed on the electronic device 400 and the screen displayed on the external electronic device 500 may be different. 9 and 10 illustrate an example of a sink device 500 (e.g., a refrigerator or the like) based on a vertical display (e.g., a vertical display fixed to the longitudinal direction), but the various embodiments are not limited thereto . For example, according to various embodiments, the display may operate in a display-based sink device 500 (e.g., a tablet PC, etc.) that is used to rotate the display in the landscape orientation in the primary direction.

9, an electronic device 400 (hereinafter, referred to as a source device) for reproducing a content in a portrait layout and having a content resolution of 1920x1080, as shown in FIG. 9 and FIG. 9, An external electronic device 500 (hereinafter referred to as a sink device) (for example, a refrigerator) in which the display 750 has a resolution of 1920 x 1080 and the display type is a vertical display 750 may be mirror-cast to each other. According to various embodiments, the source device 400 may determine through the negotiation procedure with the sink device 500 that the display type of the sink device 500 is the vertical display 750 and the supported resolution is 1920x1080 . For example, the sink device 500 may provide resolution information (e.g., 1920x1080) and angle information (e.g., rotation 270) in which the display is rotated in the horizontal direction. The source apparatus 400 can determine whether the display type of the sink device 500 is the vertical direction or the horizontal direction based on the horizontal resolution information and the rotated angle information.

The source apparatus 400 may generate data (e.g., reproduced data) to be transmitted to the sink apparatus 500 so as to correspond to the determined resolution (e.g., 1920x1080). For example, the source apparatus 400 can generate the shared data by changing the reproduction data having the specific resolution (1920x1080) to correspond to the resolution (1920x1080) of the sink apparatus 500. [

According to various embodiments, the source device 400 may generate shared data for a vertical display, based on the display type of the sink device 500 being a vertical display 750. [ For example, as shown in the example, the source device 400 rotates a virtual display 710 (e.g., a canvas) for drawing (or creating) shared data when generating shared data It can be changed into a vertical form. According to one embodiment, the source device 400 may generate a virtual display 710 with a resolution of 1920x1080 corresponding to the resolution (1920x1080) of the sink device 500 when generating the shared data. According to various embodiments, the source apparatus 400 may generate the vertical reproduction data of the example in a vertical and horizontal aspect ratio (9:16) within the virtual display 710 in the vertical form. For example, the source apparatus 400 can be generated in the form of vertical reproduction data (e.g., portrait layout) displayed on the source apparatus 400 by using the virtual display 710 in the vertical form. According to various embodiments, the source apparatus 400 may generate shared data without rotation of the reproduction data, for example, < 90 > The source device 400 may generate the shared data using the virtual display 710 corresponding to the supported resolution of the sink device 500. [

According to various embodiments, the source apparatus 400 may rotate the virtual display 710 in the default horizontal form to a vertical form (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 of the sink device 500 Rotation). In various embodiments, the source device 400 may determine the rotation and / or the direction of rotation of the virtual display 710 based on the type information of the display determined in the negotiation procedure. The source device 400 may use the rotated virtual display 710 to generate shared data to be displayed on the vertical display 750.

According to one embodiment, the virtual display 710 may be provided in a landscape format to correspond to a display of essentially landscape orientation. The source device 400 rotates vertically (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 (e.g., rotation angle information) of the sink device 500, Can also be rotated).

In accordance with various embodiments, the source apparatus 400 may provide the shared data generated by the virtual display 710, corresponding to the vertical display 750, to the sink device 500, as shown, for example, (E.g., a frame buffer) for transmitting the image frame data to the buffer 730. [ The source device 400 may transmit the image frame data to the sink device 500 via wireless communication.

In accordance with various embodiments, the sink device 500 may immediately display the image frame data received from the source device 400, as shown in the example. For example, as image frame data aligned in the source device 400 to correspond to the vertical display 750 is transmitted, the sink device 500 may perform some processing (e.g., rotation and / or scaling, etc.) on the received image The received image frame data can be displayed immediately. For example, the same type of data as the data (e.g., see FIG. 903) that the source device 400 is drawn through the virtual display 710 may be displayed via the vertical display 750.

According to various embodiments, the source apparatus 400 may be configured to transmit the reproduction data reproduced in the source apparatus 400 to the sink apparatus 500, sequentially or in parallel, as shown in the example , And the user interface 900 related to the control of the reproduction data can be displayed on the source device 400 according to the presentation mode. The user interface 900 can be implemented in various forms according to the type of reproduction data (e.g., game data, document data, moving picture data, etc.).

10, there is shown an electronic device 400 (hereinafter, referred to as a source device) that reproduces content in a landscape layout and has a content resolution of 1920x1080, as shown in an example <1001> and an example <1007> , The external electronic device 500 (hereinafter referred to as a sink device) (e.g., a refrigerator) in which the display 750 has a resolution of 1920 x 1080 and the display type is the vertical display 750 may be mirror-cast to each other. According to various embodiments, the source device 400 may determine through the negotiation procedure with the sink device 500 that the display type of the sink device 500 is the vertical display 750 and the supported resolution is 1920x1080 . For example, the sink device 500 may provide resolution information (e.g., 1920x1080) and angle information (e.g., rotation 270) in which the display is rotated in the horizontal direction. The source apparatus 400 can determine whether the display type of the sink device 500 is the vertical direction or the horizontal direction based on the horizontal resolution information and the rotated angle information.

The source apparatus 400 may generate data (e.g., reproduced data) to be transmitted to the sink apparatus 500 so as to correspond to the determined resolution (e.g., 1920x1080). For example, the source apparatus 400 can generate the shared data by changing the reproduction data having the specific resolution (1290x1080) to correspond to the resolution (1920x1080) of the sink apparatus 500. [

According to various embodiments, the source device 400 may generate shared data for a vertical display, based on the display type of the sink device 500 being a vertical display 750. [ For example, as shown in the example <1003>, the source device 400 rotates a virtual display 710 (e.g., a canvas) for drawing (or creating) shared data It can be changed into a vertical form. According to one embodiment, the source device 400 may generate a virtual display 710 with a resolution of 1920x1080 corresponding to the resolution (1920x1080) of the sink device 500 when generating the shared data. According to various embodiments, the source apparatus 400 may generate screen data of the horizontal form of the example < 1001 > in a horizontal and vertical ratio (16: 9) in the vertical form of the virtual display 710. [ For example, the source apparatus 400 can be generated in the form of landscape data (e.g., landscape layout) displayed on the source apparatus 400 using the virtual display 710 in the form of a vertical form. According to various embodiments, the source device 400 may generate shared data without rotation of screen data, for example, < 1001 > and example < 1003 >. The source device 400 may generate the shared data to be displayed on the vertical display 750 using the virtual display 710 corresponding to the supported resolution of the sink device 500. [

According to one embodiment, the virtual display 710 may be provided in a landscape format to correspond to a display of essentially landscape orientation. The source device 400 rotates vertically (e.g., 90 degrees or 270 degrees) to correspond to the vertical display 750 (e.g., rotation angle information) of the sink device 500, Can also be rotated).

According to various embodiments, the source apparatus 400 may provide the shared data generated by the virtual display 710 corresponding to the vertical display 750 to the sink device 500, as shown in the example < RTI ID = 0.0 > (E.g., a frame buffer) for transmitting the image frame data to the buffer 730. [ The source device 400 may transmit the image frame data to the sink device 500 via wireless communication.

According to various embodiments, the sink device 500 may display the image frame data received from the source device 400 directly, as shown in the example. For example, as image frame data aligned in the source device 400 to correspond to the vertical display 750 is transmitted, the sink device 500 may perform some processing (e.g., rotation and / or scaling, etc.) on the received image The received image frame data can be displayed immediately. For example, data of the same type as the data (e.g., see example <1003>) that the source device 400 is drawn through the virtual display 710 may be displayed via the vertical display 750.

According to various embodiments, the source apparatus 400 may be configured to transmit the reproduction data reproduced in the source apparatus 400 to the sink apparatus 500, sequentially or in parallel, as shown in the example , The user interface 1000 related to the control of the reproduction data can be displayed on the source device 400 according to the presentation mode. The user interface 1000 can be implemented in various forms according to the type of reproduction data (e.g., game data, document data, moving picture data, etc.).

9 and 10, according to various embodiments, the source device 400 generates a virtual display 710 in the horizontal or vertical direction corresponding to the display type of the sink device 500 , The virtual display 710 can be generated with a resolution corresponding to the resolution of the sink device 500. Thus, according to various embodiments, the shared data generated in the source device 400 can provide, for example, direct display and effects on the corresponding directional display of the sink device 500, Can be prevented.

11 is a flow chart illustrating an operational method of providing a mirror cast service of an electronic device in accordance with various embodiments of the present invention.

As shown in FIG. 11, FIG. 11 may show an example of providing a miracast service in an electronic device 400 serving as a source device for providing (sharing) data.

Referring to FIG. 11, at operation 1101, a processor (e.g., processors 120 and 210 of FIG. 1 or 2, hereinafter referred to as processor 210) of an electronic device 400 The electronic device 500 and the mirror cast can be connected. According to various embodiments, the processor 210 may couple the miracast with the external electronic device 500 in response to a user input for the miracast service.

At operation 1103, the processor 210 may obtain device information related to the external electronic device 500 from the external electronic device 500. For example, the device information can be obtained based on the above-described operation 603 and operation 605 of FIG. According to various embodiments, the device information may include resolution information supported by the external electronic device 500 and type information of the display. According to various embodiments, the device information includes information about the mode of operation of the external electronic device 500, information about the type of external electronic device 500 (e.g., tablet PC, refrigerator, television, etc.) Information, or information of a user input back channel (UIBC), and the like.

At operation 1105, the processor 210 may analyze the acquired device information. For example, the processor 210 may parse the acquired device information to obtain information about the resolution and display type (or mode of operation mode) associated with the external electronic device 500.

At operation 1107, the processor 210 may determine an optimal resolution common to the external electronic device 500 based on the device information, and determine the display type (or operating mode direction) of the external electronic device 500.

At operation 1109, the processor 210 may determine whether the display type of the external electronic device 500 is a vertical display based on the device information. Alternatively, according to various embodiments, the processor 210 may determine whether the external electronic device 500 is operating in the longitudinal direction (e.g., the display is rotated vertically) based on the device information.

In operation 1109, if the processor 210 determines that the display type is a vertical display (operation 1109), then in operation 1111, the processor 210 may generate a vertical virtual display having the determined resolution. According to one embodiment, as described in the description with reference to FIGS. 7 to 10 above, the processor 210 may be configured to display a virtual display of a predetermined horizontal shape in a vertical (vertical) form corresponding to the vertical display of the external electronic device 500 (E.g., rotate 90 degrees or 270 degrees). According to various embodiments, the processor 210 may be operable in a manner that, even if the external electronic device 500 operable in either the horizontal or vertical direction is operating in the longitudinal direction (e.g., the display is rotated in the longitudinal direction) A virtual display of a vertical form can be generated.

At operation 1109, if the processor 210 determines that the display type is not a vertical display (NO at operation 1109), for example, it determines that it is a horizontal display, then at operation 1113, Display can be generated. According to one embodiment, the processor 210 may use a virtual display of a predetermined horizontal shape. According to various embodiments, the processor 210 may be configured to perform the operations described in operation 1113 (e. G., If the external electronic device 500 is operable in both the horizontal and vertical directions) A virtual display of a landscape form can be generated.

In operation 1115, the processor 210 may generate data corresponding to the resolution and display type of the external electronic device 500, using the virtual display generated in operation 1111 or operation 1113. [

At operation 1117, the processor 210 may transmit the generated data to the external electronic device 500 using wireless communication. According to various embodiments, the processor 210 may transmit in streaming when transmitting data.

12 is a diagram illustrating an exemplary operation for processing user input in a mirror cast service in accordance with various embodiments of the present invention.

As shown in FIG. 12, FIG. 12 may show an example of an operation for processing user input (e.g., touch) in the sink device in various embodiments.

12, in operation 1201, the electronic device 400 (hereinafter referred to as a source device) and the external electronic device 500 (hereinafter referred to as a sink device) may be in a mirror cast connection state as described above.

At operation 1203, the sink device 500 may sense user input. For example, the user can touch one area in the vertical display 750, and the sink device 500 can sense a touch by the user.

At operation 1205, sink device 500 may transmit user input (e.g., control messages) to source device 400 via a user input back channel (UIBC) upon sensing user input. According to various embodiments, the sink device 500 may control the source device 400 in response to a user input in the sink device 500, without requiring a separate input process (e.g., coordinate transformation, etc.) The coordinates of the touched point can be transmitted. According to various embodiments, in the case of a vertical display-based sink device 500 whose longitudinal direction is fixed fundamentally, since width and height have (x, y), respectively, y) coordinates. According to various embodiments, in the case of the display-based sink device 500 that is used to rotate the display in the longitudinal direction in the basic direction, since the width and the height of the sink device 500 have (y, x) , the x coordinate (eg, width) can be transformed into a y coordinate, the y coordinate (eg, height) can be transformed into an x coordinate, and the transformed (x, y) coordinate can be transferred.

In operation 1207, the source device 400 may process the received user input when a user input is received from the sink device 500. [ For example, the source device 400 may process display turn-on or content (or function) related controls based on user input.

In operation 1209, the source device 400 may transmit the input processing result to the sink device 500 when the processing of the user input is completed. Although in the various embodiments, the source device 400 shown in FIG. 10 transmits an input processing result to the sink device 500, the various embodiments are not limited thereto. For example, the source apparatus 400 may not transmit the input processing result to the sink apparatus 500. [ In this case, the user can confirm whether or not the input processing of the user input is performed through the change of the screen of the sink device 500 transmitted and received from the source device 400.

Meanwhile, although FIG. 10 shows that the sink device 500 transmits the user input through the user input back channel to control the source device 400, the present invention is not limited thereto. For example, the source device 400 may detect a user input and send a control message to the sink device 500.

13 is a flow diagram illustrating an operational method for processing user input in an electronic device in accordance with various embodiments of the present invention.

As shown in FIG. 13, FIG. 13 may show an example in which user input is received in the electronic device 400 serving as a source device for providing (sharing) data to perform related processing.

13, processor (e.g., processors 120 and 210 of FIG. 1 or 2, hereinafter referred to as processor 210) of electronic device 400 (e.g., source device) And may be in a mirror cast connection with the electronic device 500 (e.g., a sink device). According to various embodiments, the processor 210 may couple the miracast with the external electronic device 500 in response to a user input for the miracast service.

At operation 1303, the processor 210 may sense user input. For example, the processor 210 may detect an internal user input that occurs directly in the electronic device 400 or an external user input that is indirectly generated (or received) by the external electronic device 500.

At operation 1305, the processor 210 may determine whether the user input is an external user input. For example, the processor 210 may determine whether the user input is a user input received via a user input back channel (UIBC) from a miraculously connected external electronic device 500, or a user input received by the electronic device 400 Or not.

In operation 1305, if processor 210 determines that it is not an external user input (NO in operation 1305), it may proceed to operation 1311 to process the operation after operation 1311.

At operation 1305, if processor 210 determines that it is an external user input (operation 1305), at operation 1307, the type of external electronic device 500 may be determined. For example, the processor 210 may determine the type (e.g., display type, operating mode, etc.) of the external electronic device based on the device information obtained in performing the negotiation procedure in the mirror communication connection with the external electronic device 500 .

In operation 1309, the processor 210 determines whether the external electronic device 500 is a device based on a vertical display (e.g., a refrigerator with a vertical display with a longitudinally fixed base) ) (Hereinafter referred to as a first type device) or a device capable of operating in both a horizontal direction and a vertical direction (e.g., a tablet PC or the like) (hereinafter referred to as a second type device).

At operation 1309, if the processor 210 determines that the external electronic device 500 is a first type device (operation 1309), at operation 1311, the processor 210 performs data control according to user input without conversion to the received user input Can be processed. For example, in the case of a vertical display in which the basic direction is the vertical direction, since the width and the height are respectively (x, y), the point (x, y) So that the operation can be processed.

At operation 1309, if processor 210 determines that external electronic device 500 is a second type device (NO at operation 1309), at operation 1313, coordinate value conversion for user input may be performed. For example, when the display is used in a landscape orientation with the base being in the landscape orientation, the x and y coordinates (e.g., width) in the coordinates have y coordinates , And y coordinates (eg, height) can be converted to x coordinates. For example, the processor 210 may convert (y, x) coordinates into (x, y) coordinates.

At operation 1311, the processor 210 may process data control according to user input based on the transformed coordinate values. For example, the processor 210 may recognize the converted (x, y) point and process the operation therefor.

At operation 1317, the processor 210 may transmit the controlled data to the external electronic device 500 at operation 1311 or at operation 1315.

In accordance with various embodiments, FIG. 13 determines whether to convert for a user input received via a user input back channel (UIBC), depending on the type of external electronic device 500 in the electronic device 400, The coordinate values of the user input are converted based on the above-described embodiments, but the various embodiments are not limited thereto. For example, if the external electronic device 500 identifies an operating mode and transmits the transformed coordinate values accordingly, the electronic device 400 may not perform operations related to transforming the received user input. For example, operations 1305, 1307, 1309, 1313 and 1315 may be omitted in Fig.

14 is a flow chart illustrating an operational method for processing user input in an electronic device in accordance with various embodiments of the present invention.

As shown in FIG. 14, FIG. 14 may show an example of performing an associated process for user input in an external electronic device 500 operating as a sink device for receiving (sharing) data. For example, FIG. 14 illustrates an example of processing user input in an external electronic device 500 (e.g., a tablet PC) that is operable in both the horizontal and vertical directions. According to various embodiments, in FIG. 14, the external electronic device 500 may be an apparatus in which the basic operation mode is set to be in the horizontal direction.

14, a processor (e.g., processors 120 and 210 of FIG. 1 or 2, hereinafter referred to as processor 210) of an external electronic device 500 (e.g., a sink device) And may be in a mirror cast connection with the electronic device 400 (e.g., a source device). According to various embodiments, the processor 210 may couple the mirror 400 with the electronic device 400 in response to a connection request of the electronic device 400.

At operation 1403, the processor 210 may sense user input. For example, the processor 210 may sense an internal user input originating directly from the external electronic device 500.

At operation 1405, the processor 210 may determine whether the orientation of the display is in a vertical orientation. According to various embodiments, it can be assumed in Fig. 14 that the display of the external electronic device 500 is transverse to the base.

At operation 1405, the processor 210 may determine whether the external electronic device 500 is operating in the longitudinal direction.

At operation 1405, if the processor 210 determines that the external electronic device 500 is not in the vertical direction (NO at operation 1405), the processor 210 may operate based on, for example, In operation 1407, a control message according to user input may be transmitted to the electronic device 400 (e.g., a source device). According to various embodiments, processor 210 may provide control messages (e.g., (x, y) coordinates) corresponding to user input related to controlling electronic device 400 via a user input back channel (UIBC) 400). According to various embodiments, the processor 210 can transmit (x, y) coordinates as it is without processing of transforming the coordinates, since the width and the height are respectively (x, y) have.

In operation 1405, if processor 210 determines that external electronic device 500 is in the vertical orientation (e.g., operation 1405), for example, In operation 1409, coordinate values for user input may be converted. According to various embodiments, the processor 210 has (y, x) in which the width and height are reversed (y, x), respectively, when used in a portrait orientation, Eg width) to y coordinates, and y coordinates (eg height) to x coordinates. For example, the processor 210 may convert (y, x) coordinates into (x, y) coordinates.

At operation 1411, the processor 210 may send a control message according to user input to the electronic device 400 (e.g., a source device) based on the transformed coordinate values. According to various embodiments, the processor 210 may provide control messages (e.g., transformed (x, y) coordinates) corresponding to user input related to control of the electronic device 400 via a user input back channel (UIBC) To the device (400).

At operation 1413, the processor 210 may receive the changed data in response to the control message by operation 1407 or operation 1411, and may display the received data via the display.

15 and 16 are diagrams illustrating operation scenarios for handling user input in a mirror cast service in accordance with various embodiments of the present invention.

15 and 16, the electronic device 400 (hereinafter referred to as a source device) and the external electronic device 500 (hereinafter referred to as a sink device) each have a specific resolution (for example, 1920 x 1080) State. For example, the source apparatus 400 may generate data according to the resolution determined in the negotiation procedure based on the mirror apparatus connection with the sink apparatus 500, and provide the generated data to the sink apparatus 500.

Referring to FIG. 15, FIG. 15 may show an example of processing a user input in a vertical display-based sink device 500 (e.g., a refrigerator or the like). In accordance with various embodiments, the sink device 500 in FIG. 15 may be an apparatus having a vertical display fixed in the longitudinal direction by way of example.

15, when the touch input is detected at a P point (e.g., (x, y) coordinates) in a mirror cast connection with the source device 400, the sink device 500 outputs coordinates (E.g., (x, y) coordinates). The sink device 500 can transmit the (x, y) coordinates of the sink device 500 directly to the source device 400 without coordinate transformation for the touch input. For example, in the case of a vertical display in which the base is the vertical direction, since the width and height are respectively (x, y), the coordinates of the (x, y) To the device (400). According to various embodiments, when the resolution of the source device 400 and the sink device 500 are different, the coordinates (e.g., (x, y) coordinates for the P point) are corrected to match the resolution of the source device 400 Operation may be further included. Such correction may be handled by the source device 400 or the sink device 500, which is determined to be the operating entity in the negotiation procedure between the devices.

The source device 400 may perform a function corresponding to the coordinates received from the sink device 500 (e.g., (x, y) coordinates of the sink device). The source device 400 can selectively transmit data (e.g., screen data, playback data) that is changed according to the performance of the function to the sink device 500. For example, the source apparatus 400 can transmit changed data while displaying the changed data in the mirroring mode, and can transmit the changed data without presentation of changed data in the presentation mode.

Referring to FIG. 16, FIG. 16 may show an example of processing a user input in a sink device 500 (e.g., a tablet PC or the like) which is operable in both the horizontal direction and the vertical direction. In accordance with various embodiments, the sink device 500 in FIG. 16 may illustrate an apparatus in which the basic operation mode is set to be in the horizontal direction.

16, when the touch input is detected at a P point (e.g., (y, x) coordinate) in a state where the mirror device is connected to the source device 400 in a mirror cast state, (E.g., (y, x) coordinates). Sink device 500 may transform the coordinates for the touch input and transmit the (x, y) coordinates of the converted sink device to the source device 400. For example, in the case where the display device 500 is rotated in the vertical direction using the basic horizontal display, since the width and the height of the display device 500 are respectively (y, x) rotated in opposite directions, : Width) into y coordinates, and y coordinates (eg, height) into x coordinates. For example, the sink device 500 can convert the (y, x) coordinates of the touch-input P point to the (x, y) coordinates. According to various embodiments, when the resolution of the source device 400 and the sink device 500 are different, the coordinates (e.g., (x, y) coordinates for the P point) are corrected to match the resolution of the source device 400 Operation may be further included. Such correction may be handled by the source device 400 or the sink device 500, which is determined to be the operating entity in the negotiation procedure between the devices.

The source device 400 may perform a function corresponding to the coordinates received from the sink device 500 (e.g., (x, y) coordinates of the sink device). The source device 400 can selectively transmit data (e.g., screen data, playback data) that is changed according to the performance of the function to the sink device 500. For example, the source apparatus 400 can transmit changed data while displaying the changed data in the mirroring mode, and can transmit the changed data without presentation of changed data in the presentation mode. According to various embodiments, when the sink device 500 does not include the function of coordinate conversion, the coordinate conversion may be processed by the source device 400 as described in the description section with reference to Fig. 13 described above.

As described above, according to various embodiments, an operation method of an electronic device includes acquiring device information from a mirror-connected external electronic device, determining a resolution and a display type of the external electronic device based on the device information Generating data corresponding to the vertical display based on the resolution when the external electronic device is a vertical display, and transmitting the data to the external electronic device using the wireless communication unit can do.

According to various embodiments, the device information may include a resolution and a display type supported by the external electronic device. According to various embodiments, the device information includes information about an operating mode of the external electronic device, information on the type of the external electronic device, manufacturer information, model name information, or user input back channel (UIBC) As shown in FIG.

According to various embodiments, the determining may include determining whether the external electronic device is a vertical display based on the display type, or determining whether the external electronic device is operating in a vertical direction .

According to various embodiments, the generating may include determining an optimal resolution common to the external electronic device based on the device information and determining a display type of the external electronic device, Generating a vertical virtual display having the determined resolution, and generating data corresponding to the resolution and the display type of the external electronic device using the virtual display.

According to various embodiments, the generating process may include the steps of rotating the virtual display of a predetermined horizontal shape in the vertical direction to correspond to the vertical display of the external electronic device, And a step of generating the data.

According to various embodiments, the generating may include generating a horizontal virtual display having the determined resolution if the display type is a horizontal display.

Receiving coordinates from a user input through a user input back channel (UIBC) from the external electronic device according to various embodiments; determining a type of the external electronic device in response to the coordinate reception; If the electronic device is a first type device based on a vertical display, recognizing the coordinates without conversion to the coordinates, and if the electronic device is a second type device used by rotating the device in the longitudinal direction , Performing the transformation of the coordinates, and recognizing the transformed coordinates.

According to various embodiments, the step of transforming the coordinates may comprise performing the coordinate transformation by transforming the x coordinate to the y coordinate and the y coordinate to the x coordinate in the above coordinates.

According to various embodiments, the external electronic device may further include a step of detecting a user input in a state of being mirror-linked with the electronic device, a step of determining a direction of the display in response to user input detection, Converting the coordinate of the user input when the user rotates in the vertical direction in the horizontal direction, and transmitting the converted coordinate to the electronic device.

The various embodiments of the present invention disclosed in the present specification and drawings are merely illustrative examples of the present invention and are not intended to limit the scope of the present invention in order to facilitate understanding of the present invention. Accordingly, the scope of the present invention should be construed as being included in the scope of the present invention, all changes or modifications derived from the technical idea of the present invention.

400: electronic device (source device)
120, 210: Processor
500: External electronic device (sink device)
710: Virtual display

Claims (20)

In an electronic device,
A wireless communication unit for a miracast connection and data transmission / reception;
A display for displaying data; And
And a processor operatively coupled to the wireless communication unit and the display,
Acquiring device information from a mirror-connected external electronic device,
Determining a resolution and a display type of the external electronic device based on the device information,
If the external electronic device is a vertical display, generating data corresponding to the vertical display based on the resolution, and
And to transmit the data to the external electronic device using the wireless communication unit.
2. The apparatus of claim 1,
Wherein the external device is configured to include a resolution and a display type supported by the external electronic device.
3. The method according to claim 2,
Information on an operation mode of the external electronic device, information on a type of the external electronic device, manufacturer information, model name information, or information on a user input back channel (UIBC).
3. The apparatus of claim 2,
And determine whether the external electronic device is a vertical display based on the display type or whether the external electronic device is operating in a vertical direction.
2. The apparatus of claim 1,
Determining an optimum resolution common to the external electronic device based on the device information, determining a display type of the external electronic device,
Generating a vertical virtual display having the determined resolution if the display type is a vertical display, and
And use the virtual display to generate data corresponding to the resolution and display type of the external electronic device.
6. The apparatus of claim 5,
To rotate the virtual display of the default horizontal shape in the vertical direction to correspond to the vertical display of the external electronic device, and to generate data using the rotated vertical display of virtual display.
6. The apparatus of claim 5,
And to generate a horizontal virtual display having the determined resolution if the display type is a landscape display.
2. The apparatus of claim 1,
Receiving coordinates from the external electronic device via a user input back channel (UIBC) according to user input,
Determining a type of the external electronic device in response to the coordinate reception,
If the external electronic device is a first type device based on a vertical display, recognizing the coordinates without conversion to the coordinates, and
And if the external electronic device is a second type device that rotates in the longitudinal direction in the predetermined horizontal direction and uses it, recognizes the transformed coordinates.
9. The apparatus of claim 8,
Wherein in said coordinates, the x coordinate is converted into a y coordinate and the y coordinate is converted into an x coordinate to perform a coordinate transformation.
The electronic device according to claim 1,
Detecting a user input in a mirror cast connection with the electronic device,
In response to user input detection, the direction of the display is determined,
If the orientation of the display is in a state of being rotated and used in a longitudinal direction in a preferred horizontal direction, transforming the coordinates for the user input, and
And transmit the transformed coordinates to the electronic device.
A method of operating an electronic device,
Acquiring device information from a mirror-connected external electronic device,
Determining a resolution and a display type of the external electronic device based on the device information,
If the external electronic device is a vertical display, generating data corresponding to the vertical display based on the resolution; and
And transmitting the data to the external electronic device using the wireless communication unit.
12. The method according to claim 11,
And a resolution and a display type supported by the external electronic device.
13. The method according to claim 12,
Information on an operating mode of the external electronic device, information on a type of the external electronic device, manufacturer information, model name information, or information on a user input back channel (UIBC).
13. The method of claim 12,
Determining whether the external electronic device is a vertical display based on the display type or whether the external electronic device is operating in a vertical direction.
12. The method of claim 11, wherein the generating comprises:
Determining an optimal resolution common to the external electronic device based on the device information and determining a display type of the external electronic device;
Generating a vertical virtual display having the determined resolution when the display type is a vertical display; and
And generating data corresponding to the resolution and the display type of the external electronic device using the virtual display.
16. The method of claim 15,
Rotating the virtual display of the default horizontal shape in the vertical direction so as to correspond to the vertical display of the external electronic device, and
And generating data using a virtual display in the rotated longitudinal direction.
16. The method of claim 15,
And generating a horizontal virtual display having the determined resolution if the display type is a landscape display.
12. The method of claim 11,
Receiving coordinates from the external electronic device via a user input back channel (UIBC) according to user input;
Determining a type of the external electronic device in response to the coordinate reception;
If the external electronic device is a first type device based on a vertical display, recognizing the coordinates without conversion to the coordinates, and
And if the external electronic device is a second type device that rotates in a vertical direction in a predetermined horizontal direction, performing the conversion of the coordinates and recognizing the converted coordinates.
19. The method of claim 18,
Wherein in said coordinates, the x-coordinate is transformed into a y-coordinate, and the y-coordinate is transformed into an x-coordinate to perform coordinate transformation.
12. The electronic device according to claim 11,
Detecting a user input when the electronic device is connected to a mirror cast,
Determining a direction of the display in response to user input detection,
Converting the coordinates of the user input when the direction of the display is rotated in a vertical direction in a predetermined horizontal direction and used;
And transmitting the transformed coordinates to the electronic device.

Images