JP2014222432A - Adapter device, mirroring system, and mirroring method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a system that mirrors a reproduced screen of a terminal to a display to cope with a plurality of mirroring standards inexpensively.SOLUTION: An adapter device 2 includes: a relay side communication unit 21 for establishing radio connection with a mobile terminal 1; a mode selection unit for accepting input of a connection waiting mode of the relay side communication unit 21; a communication setting unit 22 for setting a communication standard of the connection waiting mode accepted by the mode selection unit to the relay side communication unit 21 as a communication standard of radio connection that the relay side communication unit 21 can accept; and a rendering unit 27 for drawing output screen data from the mobile terminal 1 having received via radio connection of the relay side communication unit 21, and for transferring the drawn result to a connected display device 4.

Description

  The present invention relates to a technology of an adapter device, a mirroring system, and a mirroring method.

  Mirroring is a technique for transferring and displaying display data from a generation device that generates display data such as a smartphone to a display device such as a liquid crystal display. Thus, not only the user of the generation side device but also the surrounding friends can view the same screen of the display side device, so that movie viewing and presentation can be realized easily (see Patent Documents 1 and 2). .

  Note that, in the case where presentations are given sequentially in a meeting, a plurality of generation-side devices may want to transfer display data to a single display-side device, respectively. In view of this, a method has been proposed in which the display side device selects which display data is permitted from among a plurality of generation side devices that have requested display (see Patent Documents 3 and 4). . By these methods, the display data for one device is displayed while occupying one display-side device without causing the display data to collide with each other.

JP 2012-177902 A JP 11-296270 A JP 2012-160020 A JP 2005-266777 A

  In the case where a plurality of generation side devices share one display side device, it is necessary to match not only the display data selection process but also the generation side device mirroring standard and the display side device mirroring standard. . Examples of the mirroring standard include Miracast (registered trademark), AirPlay (registered trademark), and DLNA (Digital Living Network Alliance) (registered trademark).

  AirPlay is a standard provided by Apple and operates on iOS devices such as iPad (registered trademark). Miracast and DLNA are standards that run on Android, a Linux (registered trademark) -based platform. Miracast is a new standard that comes standard with Android (registered trademark) 4.2, and DLNA is an older standard than Miracast. In these standards, the preparations for transmitting display data from the display data transmission side to the reception side are different from each other (for example, establishment of a communication channel). Can be sent.

Therefore, when mirroring is performed using a display-side device that does not comply with the mirroring standard of the generation-side device, it is necessary to prepare an adapter device that mediates between the generation-side device and the display-side device. If this adapter device complies with the mirroring standard of the generation side device, the display data received by the adapter device according to the mirroring standard is transferred to the display side device as it is as a displayable format, so that the display side device Data display is possible. That is, the adapter device absorbs the mirroring standard that is not supported by the display side device.
However, if adapter devices for each of the three standards are prepared, the cost of money and the space are imposed, and the image input terminal of the display side device is filled by three inputs, resulting in poor expandability. .

  Therefore, the main object of the present invention is to support a plurality of mirroring standards at a low cost in a system for mirroring a playback screen of a terminal to a display.

In order to solve the above-described problem, the adapter device of the present invention provides:
A mode selection unit that accepts an input of a communication standard among a plurality of communication standards as a connection waiting mode;
A communication setting unit that sets the communication standard of the connection waiting mode received by the mode selection unit to itself as a communication standard of an acceptable wireless connection;
A relay side communication unit that establishes wireless connection with the communication standard of the connection waiting mode set by the communication setting unit with a mobile terminal;
Draw output screen data from the portable terminal received via the wireless connection of the relay side communication unit, and transfer the drawing result to the connected display device, thereby displaying the drawing result on the display device And a rendering unit.
Other means will be described later.

  ADVANTAGE OF THE INVENTION According to this invention, in the system which mirrors the reproduction | regeneration screen of a terminal to a display, it can be made to respond | correspond to several mirroring standards at low cost.

It is a block diagram which shows the mirroring system regarding one Embodiment of this invention. It is a block diagram which shows the detail of the mirroring system regarding one Embodiment of this invention. It is process explanatory drawing which shows operation | movement of the mirroring system regarding one Embodiment of this invention. It is explanatory drawing which shows the output screen of the adapter apparatus regarding one Embodiment of this invention. It is explanatory drawing which shows the output screen of the portable terminal regarding one Embodiment of this invention.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a configuration diagram showing a mirroring system. The mirroring system includes a mobile terminal 1 (1a to 1c), an adapter device 2, and a display device 4.
On the display device 4, display data that is the content that the mobile terminal 1 displays on its own display screen (such as a liquid crystal screen of a touch panel) is mirrored (transferred) and displayed. The adapter device 2 receives the display data of the mobile terminal 1 and transfers it to the display device 4.
As a communication means between the devices constituting the mirroring system, for example, wireless communication such as wi-fi (registered trademark) which is a logical communication path formed between the adapter device 2 and the portable terminal 1, an adapter device Wired communication such as a high-definition multimedia interface (HDMI) (registered trademark) cable that connects the 2 and the display device 4.

  1 is configured as a computer having a CPU (Central Processing Unit), a memory, a hard disk (storage means), and a communication interface. In this computer, the CPU operates each processing unit by executing a program read on the memory.

Here, the portable terminal 1 of various mirroring standards is connected to the adapter device 2. In FIG. 1, three standards such as Miracast, AirPlay, and DLNA are illustrated as those standards, but are not limited to these three standards.
Note that one mirroring standard corresponds to one or more wireless standards (uses the wireless standards) in order to establish a communication path for performing mirroring. For example, the Miracast mirroring standard stipulates that the Wi-Fi-Direct mode communication standard be used. On the other hand, the mirroring standards of AirPlay and DLNA stipulate that the Wi-Fi-infrastructure mode communication standard is used.
Therefore, in the mirroring system of FIG. 1, the mobile terminal 1 a is wirelessly connected to the adapter device 2 using the Wi-Fi-Direct mode communication standard defined by Miracast.
The portable terminal 1b wirelessly connects to the adapter device 2 using a Wi-Fi-infrastructure mode communication standard defined by AirPlay.
The portable terminal 1c wirelessly connects to the adapter device 2 using a Wi-Fi-infrastructure mode communication standard defined by DLNA.

FIG. 2 is a configuration diagram showing details of the mirroring system.
The mobile terminal (Miracast) 1a includes a terminal-side display unit 11a, a rendering unit 12a, an encoding unit 13a, and a terminal-side communication unit 14a.
The rendering unit 12a draws (renders) display data of a screen to be displayed on the terminal side display unit 11a such as its own touch panel. The encoding unit 13a creates encoding data by performing real-time encoding processing on the display data of the rendering unit 12a, and transmits the encoded data to the adapter device 2 via the terminal side communication unit 14a.
The terminal-side communication unit 14a forms a logical connection in Wi-Fi-Direct mode with the relay-side communication unit 21a of the adapter device 2, and transmits the generated encoded data using the logical connection. .

The portable terminal (AirPlay) 1b includes a terminal side display unit 11b, a rendering unit 12b, an encoded data storage unit 13b, and a terminal side communication unit 14b.
The rendering unit 12b generates display data of a screen to be displayed on the terminal side display unit 11b such as its own touch panel by rendering (rendering) after decoding the encoded data in the encoded data storage unit 13b.

  The encoded data is data that has been compression-encoded (encoded) in accordance with a video file standard such as MPEG (Moving Picture Experts Group) -4. In the video file, a video codec and an audio codec are collected in a container. The decoding process is a process of analyzing the encoded data and converting it into a form that can be reproduced.

Note that the portable terminal (AirPlay) 1b may store the encoded data itself in the encoded data storage unit 13b, and holds a link to the encoded data (URL (Uniform Resource Locator) to the outside). Also good.
The terminal-side communication unit 14b is connected to the relay-side communication unit 21b of the adapter device 2 so as to be communicable in the Wi-Fi-infrastructure mode, and the encoded data in the encoded data storage unit 13b is transferred using the connection. Send.

  The portable terminal (DLNA) 1c includes a terminal side display unit 11c, a rendering unit 12c, an encoded data storage unit 13c, and a terminal side communication unit 14c. Note that the mobile terminal (DLNA) 1c and the mobile terminal (AirPlay) 1b have equivalent components whose code ends correspond to “b” and “c”. For example, the encoded data storage unit 13b corresponds to the encoded data storage unit 13c.

The adapter device 2 includes a relay side communication unit 21 (21a to 21c), a communication setting unit 22, a decoding unit 26, a rendering unit 27, and a WAN (Wide Area Network) communication unit 28.
The relay side communication unit 21 (21a to 21c) is a wireless communication unit that forms a communication path with a wireless standard corresponding to the mirroring standard with the corresponding terminal side communication unit 14 (14a to 14c).

The communication setting unit 22 sets which standard is currently used among the relay side communication units 21 (21a to 21c) of each standard. That is, the communication setting unit 22 enables only one relay-side communication unit 21 to be connected from among the relay-side communication units 21a to 21c, and disables the other two relay-side communication units 21.
That is, one adapter device 2 includes the relay side communication unit 21 for each of the three mirroring standards, and enables the relay side communication unit 21 having the same standard as the mirroring standard of the mobile terminal 1 to be connected. Thus, mirroring can be performed with any mirroring standard portable terminal 1. The validation of the mirroring standard means that the adapter device 2 is wirelessly connected using the connection information (details will be described later) of the mirroring standard.
In addition, since the mirroring standards match between the adapter device 2 and the mobile terminal 1 of the connection partner, both devices may be operated by the same user, or even between different users, The same mirroring standard may be determined in advance.

  The communication setting unit 22 disables two when the three relay side communication units 21 are used exclusively. However, when there are three wireless communication modules of the relay side communication unit 21, That is, when the three relay-side communication units 21 can be used simultaneously in parallel, it is not necessary to perform the invalidation process of the relay-side communication unit 21 different from the one to be connected. Thereby, since the Wi-Fi connection of the relay side communication unit 21 different from that to be connected is not disconnected, it is not necessary to repeat the Wi-Fi connection and the Wi-Fi connection.

The decoding unit 26 decodes the encoded data received by the relay side communication unit 21 and passes it to the rendering unit 27. In addition, when the data received by the relay-side communication unit 21 is not the encoded data itself but is a link to the encoded data, the decoding unit 26 uses the media server 31 located at the Internet link destination via the WAN communication unit 28. The encoded data in the encoded data storage unit 32 is loaded (downloaded or received by streaming) and then decoded.
The rendering unit 27 creates display data to be output to the display device 4 by rendering the display data decoded according to the standard (resolution, etc.) of the display device 4, and outputs the display data to the display device 4 via the HDMI cable. To do.

  FIG. 3 is an explanatory diagram of processing showing the operation of the mirroring system. In addition, in FIG. 3, in addition to the components of the mirroring system (adapter device 2, portable terminal 1), the operation of the user of the mirroring system is also described, but this description is for the sake of easy understanding. The mirroring system is not configured to include the user who operates the mirroring system.

  As S111, when the user presses a mode selection button (mode selection unit, not shown) of the adapter device 2, the adapter device 2 updates its connection waiting mode (S112). Alternatively, the user may activate the update process of S112 by causing the mobile terminal 1 to output a mode selection signal by infrared communication or the like. Therefore, the adapter device 2 has mode selection means (not shown) for performing the process of S112.

  In addition, every time the connection selection mode is updated, for example, the mode selection means updates the connection connection mode to “AirPlay mode”. A screen that prompts the user to confirm whether or not the update is possible may be presented, and the connection waiting mode may be updated when confirmation from the user is obtained. As a result, it is possible to prevent the connection wait mode currently used from being changed to another connection wait mode.

As S113, the adapter device 2 creates a screen indicating the connection waiting mode updated in S112, and displays the screen on the display device 4. The screen of S113 is not a screen mirrored from the mobile terminal 1, but is a screen created by the adapter device 2 itself. Therefore, the adapter device 2 includes a connection information providing unit (not shown) for performing the process of S113.
Note that the connection waiting mode is a mode in which the communication setting unit 22 sets which of the three relay side communication units 21 to use. For example, the connection waiting mode is “AirPlay mode”. Then, the communication setting unit 22 uses (actives) the relay side communication unit 21b.

In this embodiment, the following four are illustrated as connection waiting modes.
Miracast PIN (Personal Identification Number) mode: This mode waits for connection from the mobile terminal (Miracast) 1a, and PIN, which is a personal identification number, is used as the connection information (see FIG. 4A).
Miracast PBC (Push Button Configuration) mode: a mode for waiting for connection from the mobile terminal (Miracast) 1a, and using the MAC (Media Access Control) address of the adapter device 2 as connection information (FIG. 4B) reference).
In the Miracast PIN mode and Miracast PBC mode, the adapter device 2 does not actively transmit the SSID for Wi-Fi-Direct connection, and refers to the SSID requested from the mobile terminal (Miracast) 1a. It is desirable to operate in stealth mode from the viewpoint of preventing unauthorized connection.

AirPlay mode: This mode waits for connection from the portable terminal (AirPlay) 1b, and uses SSID (Service Set Identifier) for identifying the AP (Access Point) of the adapter device 2 as the connection information (FIG. 4 (c) )reference). The adapter device 2 also operates as a wireless LAN (Local Area Network) AP for providing the mobile terminal 1 with an Internet connection.
DLNA mode: A mode for waiting for a connection from the portable terminal (DLNA) 1c. As the connection information, in addition to the SSID described in the AirPlay mode, the IP address of the adapter device 2 that is the DLNA connection partner is used. (IP address of WFD-HDMI) is used (see FIG. 4D). Note that the screen of S113 includes not only the connection information but also other information (for example, a URL for logging in to the setting screen of the adapter device 2 shown in FIG. 4D and a QR (Quick Response) indicating the URL). Information for setting such as a code) may be displayed.

Regarding the update processing of the four connection waiting modes described above, when the mode selection button of S111 exists for each mode (here, four), the adapter device 2 enters the connection waiting mode corresponding to the button pressed this time. Update (S112).
Alternatively, when there is only one mode selection button in S111, the adapter device 2 has a predetermined mode order (for example, Miracast PIN mode → Miracast PBC mode → AirPlay mode → DLNA mode → Miracast PIN mode → Miracast PBC mode → ... ) To update to the next connection waiting mode as seen from the previous mode (S112).

  As S114, the user can grasp the current connection waiting mode of the adapter device 2 and its connection information by viewing the screen displayed on the display device 4 in S113.

As S <b> 121, the user inputs the connection information grasped in S <b> 114 to the mobile terminal 1. Here, the input operation may be such that new connection information is input to the mobile terminal 1 as a new setting, or a list of connection information already input to the mobile terminal 1 is presented to the user. You may choose from. The portable terminal 1 receives the input connection information (S122).
As S123, the mobile terminal 1 requests the adapter device 2 to connect based on the connection information received in S122. The connection here is the Wi-Fi-Direct connection shown in FIG. 2 or the Wi-Fi-infrastructure connection.

As S124, the adapter device 2 accepts the connection request of S123. Then, the adapter device 2 establishes a connection with the mobile terminal 1 based on the connection information.
As described in connection information in the connection waiting mode, the establishment of the connection in S124 is not only the establishment of the communication path used by the mirroring standard, but also the mirroring standard between the portable terminal 1 and the adapter device 2. Any preparatory processing may be included for future availability. The preparatory processing includes, for example, user authentication processing of the mobile terminal 1 performed through an established communication path, and startup processing of various programs for realizing the mirroring standard.
Here, the adapter device 2 accepts the connection request (S123) for the connection information according to the latest connection wait mode updated in S112, but refuses not to accept connection requests in other connection wait modes. Also good. Thereby, it is possible to specify which mobile terminal 1 is given priority for screen output by adopting one connection request from connection requests generated at the same time from a plurality of mobile terminals 1 having different standards.

  And the portable terminal 1 starts the application (mirroring application) which transfers an output screen via the connection of S124. Examples of this application include “Miracast” for the mobile terminal (Miracast) 1a, “AirPlay” for the mobile terminal (AirPlay) 1b, and “AirFun” for the mobile terminal (DLNA) 1c. AirFun is not a standard application, so it must be downloaded and installed in advance from the app download site (google play).

As S131, the user inputs a display operation (details will be described later in FIG. 5) for determining the contents of display data to be drawn on the display device 4 via the adapter device 2. The portable terminal 1 receives the input display operation (S132).
As S133, the mobile terminal 1 (mirroring application) creates display data (encoded data) based on the display operation received in S132, and transmits it to the adapter device 2 via the connection established in S123 and S124. For example, in the case of the mobile terminal (Miracast) 1a, display data is created by the encoding unit 13 encoding the display data to be displayed on its terminal side display unit 11 in real time based on the display operation. On the other hand, in the case of the portable terminal (AirPlay) 1b or the portable terminal (DLNA) 1c, the encoded data stored in its own encoded data storage units 13b and 13c (or the encoding that can be acquired from the link destination of the stored encoded data) Data specified by the display operation is read out.
In S <b> 134, the adapter device 2 extracts the display data by decoding the encoded data received in S <b> 133, and transfers the display data to the display device 4 to cause the display device 4 to output a screen.

  FIG. 5 is an explanatory diagram showing an output screen of the portable terminal output in S134. In FIG. 5, as an example of the output screen, a screen of a video playback application that operates on the mobile terminal (AirPlay) 1b is shown. The output screen is composed of an upper header area 92 and a display area 91 located therebelow.

In the header area 92, a completion button 92a, a reproduction time display column 92b, a reproduction position slide bar 92c, and a remaining time display column 92d are arranged in order from the left in order for the user to input the display operation of S131.
When the completion button 92a is pressed (touched in the case of a touch panel), the video playback application ends and the screen returns to the application selection screen (not shown).
The reproduction time display field 92b displays the reproduction time (reproduction position) of the current moving image.
When the playback position slide bar 92c is slid, the playback position is changed.
In the remaining time display field 92d, the remaining time of the current moving image is displayed. That is, the time length of the moving image is the sum of the reproduction time and the remaining time.

The display area 91 displays the content of the decoded encoded data (for example, content data such as a sport game). An operation area 93 is displayed (overlapping) on the display area 91, and a display destination area 94 is further displayed on the operation area 93.
The operation area 93 includes a group of buttons (previous item moving button 93a, pause button 93b, next item moving button 93c) for controlling playback of moving images in order from the left in order to allow the user to input the display operation of S131, and a display. A destination selection button 93d is arranged. When the display destination selection button 93d is pressed, the display destination area 94 is displayed.

The display destination area 94 is an area for selecting the display destination of the output screen currently being shown to the user.
When the local output selection field (iPad) 94a is selected (checked) by pressing, an output screen is displayed on the terminal side display unit 11 (touch panel) mounted on the portable terminal (AirPlay) 1b.
When the remote output selection field (abb) 94b is selected (checked) by pressing, an output screen is displayed on the display device 4.
In addition, regarding the screen output of local output or remote output, only one local output or remote output may be selected, or local output and remote output may be performed simultaneously. The term “mirroring” in the present embodiment is used in a broad sense of transferring an output screen, and is not limited to a system (in a narrow sense) that performs local output and remote output simultaneously.

According to the present embodiment described above, the adapter device 2 accepts connection (S124) in the connection waiting mode (S112) of each of the plurality of mirroring standards, so that the portable terminal 1 conforms to any one of the plurality of mirroring standards. And the adapter device 2 can be established.
Furthermore, the setting information corresponding to the current setting of the connection waiting mode is presented to the user on the screen before connection (S113), so that the user inputs the setting information such as SSID to the portable terminal 1 as it is. The mirroring system can be easily used without knowing details of the plurality of mirroring standards (S121).
And the adapter apparatus 2 of this embodiment respond | corresponds to multiple (for example, three) standards, respectively, compared with the structure which prepares only the number of standard adapter apparatuses corresponding to only one standard. Since components (such as the wireless antenna of the relay-side communication unit 21 and the HDMI terminal used for connection to the display device 4) can be integrated into one, it is possible to introduce a low-cost and highly scalable mirroring system. it can.

DESCRIPTION OF SYMBOLS 1 Mobile terminal 11 Terminal side display part 12 Rendering part 13 Encoding part 13 Encoded data 14 Terminal side communication part 2 Adapter apparatus 21 Relay side communication part 22 Communication setting part 26 Decoding part 27 Rendering part 28 WAN communication part 31 Media server 32 Encoded data 4 display devices

Claims (7)

A mode selection unit that accepts an input of a communication standard among a plurality of communication standards as a connection waiting mode;
A communication setting unit that sets the communication standard of the connection waiting mode received by the mode selection unit to itself as a communication standard of an acceptable wireless connection;
A relay side communication unit that establishes wireless connection with the communication standard of the connection waiting mode set by the communication setting unit with a mobile terminal;
Draw output screen data from the portable terminal received via the wireless connection of the relay side communication unit, and transfer the drawing result to the connected display device, thereby displaying the drawing result on the display device An adapter unit, comprising: a rendering unit; The adapter device further displays, on the display device, identification information of the adapter device itself as a connection destination as connection information for the mobile terminal necessary for the relay communication unit to establish a wireless connection. The adapter device according to claim 1, further comprising a connection information providing unit that performs connection. The adapter device according to claim 1, wherein the mode selection unit receives a mode selection signal indicating a communication standard of the connection waiting mode from the portable terminal as an input of the connection waiting mode. The adapter device further includes a WAN communication unit serving as a communication interface with a WAN (Wide Area Network),
4. The relay-side communication unit operates as an access point device that provides access to the WAN via the WAN communication unit for the portable terminal that has established a wireless connection. The adapter apparatus of any one of Claims. The adapter device further converts the output screen data that has undergone compression encoding encoding processing into a form that can be analyzed and reproduced when the relay-side communication unit receives the output screen data, and the rendering unit The adapter device according to any one of claims 1 to 4, further comprising: a decoding unit that notifies the device. It has the adapter apparatus of any one of Claim 1 thru | or 5, and the said display apparatus, The said adapter apparatus displays the output screen data from the said portable terminal on the said display apparatus. Mirroring system characterized by that. An adapter device that relays between the mobile terminal and the display device,
For wireless connection for receiving the output screen data of the mobile terminal, accepting an input of a connection waiting mode indicating a communication standard of a currently accepted wireless connection among a plurality of communication standards,
The communication standard of the accepted connection waiting mode is set to itself as a communication standard of acceptable wireless connection, and the set wireless connection is established with the mobile terminal,
Drawing the output screen data from the mobile terminal received via a wireless connection, and transferring the drawing result to the connected display device, thereby displaying the drawing result on the display device Mirroring method.

Images