JP6503558B2 - INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING SYSTEM, AND INFORMATION PROCESSING METHOD - Google Patents

Description

  The present disclosure relates to an information processing device, an information processing system, and an information processing method.

  Today, various types of electronic devices have the ability to encode and transmit or decode and play multimedia content. The encoded content is transmitted over the multimedia network and played back by the device that received the content. Conventionally, a multimedia network has often been formed using a wired communication technology such as HDMI (High-Definition Multimedia Interface). However, as a result of the rapid development of wireless communication technology in recent years, it has become possible to utilize a wireless communication network such as a wireless LAN (Local Area Network), for example, to transmit multimedia contents.

  A device that encodes content as needed and sends it to the network is generally referred to as a source device. In contrast, a device that receives and plays content from a source device is called a sink device. A Wi-Fi display, for example, is one of the techniques for displaying content transmitted from a source device on a sink device.

  Non-Patent Document 1 is a technical specification of a Wi-Fi display formulated by the Wi-Fi Alliance. The Wi-Fi display is also called Wi-Fi CERTIFIED Miracast (registered trademark, hereinafter also referred to simply as Miracast). The Wi-Fi display adopts Wi-Fi Direct (registered trademark) for forming a wireless communication network between wireless LAN devices without using an access point as a basic technology, and sinks high-definition video content from a source device Define a protocol for transmission to the device.

“Wi-Fi Display Technical Specification Version 1.0.0”, Wi-Fi Alliance Technical Committee, Wi-Fi Display Technical Task Group, August 24, 2012.

  However, in the technology disclosed in the above-mentioned Non-Patent Document 1, the technology related to playback control of content is mainly defined, and it is developed as to how the source device and the sink device work in cooperation with user operation. On the way. Thus, the present disclosure proposes a novel and improved information processing apparatus, an information processing system, and an information processing method that can improve the technology relating to the cooperation of a source device and a sink device with respect to user operation.

  According to the present disclosure, a wireless communication unit that performs wireless communication with another information processing apparatus, a display unit that displays an image received from the other information processing apparatus by the wireless communication unit, and the display unit An acquisition unit for acquiring an operation to a region indicated by control information received from the other information processing apparatus by the wireless communication unit among the images displayed by the display unit; and the operation according to the operation acquired by the acquisition unit An information processing apparatus is provided, comprising: a control unit configured to perform processing indicated by control information.

  Further, according to the present disclosure, a wireless communication unit performing wireless communication with another information processing apparatus, and an image is transmitted to the other information processing apparatus via the wireless communication unit, and the other information processing is performed. Control information for enabling processing to be performed when detecting an operation to a designated area of the image displayed by the apparatus is transmitted to the other information processing apparatus via the wireless communication unit. An information processing apparatus is provided, comprising: a control unit to transmit.

  Further, according to the present disclosure, there is provided a first wireless communication unit including a transmitting device and a receiving device, the transmitting device performing wireless communication with the receiving device, and the first wireless To make it possible to transmit an image to the reception side device via a communication unit, and to perform processing corresponding to a case where an operation to a designated area of the image displayed by the reception side device is detected. And a first control unit that transmits control information to the receiving device via the first wireless communication unit, and the receiving device performs wireless communication with the transmitting device. A second wireless communication unit, a display unit for displaying the image received from the transmission side device by the second wireless communication unit, and the second wireless communication among the images displayed by the display unit To the area indicated by the control information received from the transmitting device by the Having an acquisition unit for acquiring operation, and a control unit for controlling so as to perform processing indicated by the control information according to the acquired operation by the acquisition unit, the information processing system is provided.

  Further, according to the present disclosure, performing wireless communication with another information processing apparatus, displaying an image received from the other information processing apparatus, and the other of the displayed images Acquiring an operation to the area indicated by the control information received from the information processing apparatus, and controlling the information processing apparatus to perform a process indicated by the control information according to the acquired operation. , An information processing method performed by a processor is provided.

According to the technology of the present disclosure, it is possible for the sink device to directly execute processing according to the user operation without requiring recognition of user operation by the source device and instruction to the sink device.
Note that the above-mentioned effects are not necessarily limited, and, along with or in place of the above-mentioned effects, any of the effects shown in the present specification, or other effects that can be grasped from the present specification May be played.

BRIEF DESCRIPTION OF THE DRAWINGS It is explanatory drawing for demonstrating the outline | summary of the communication system which concerns on 1st Embodiment. It is a block diagram showing an example of composition of a sink device concerning a 1st embodiment. It is a figure for demonstrating a command area | region and command information. It is a block diagram showing an example of composition of a source device concerning a 1st embodiment. It is a figure which shows an example of the flow of the basic operation processing performed by the sink apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the flow of the basic operation processing performed by the source device which concerns on 1st Embodiment. It is a figure which shows an example of the flow of the call-in process performed by the sink device which concerns on 1st Embodiment, and a calling-out process. It is a figure which shows an example of the flow of the call-in process performed by the sink device which concerns on 1st Embodiment, and a calling-out process. It is a figure which shows an example of the flow of the incoming call process performed by the source device which concerns on 1st Embodiment, and an outgoing call process. It is a figure which shows an example of the flow of the incoming call process performed by the source device which concerns on 1st Embodiment, and an outgoing call process. It is a sequence diagram which shows an example of the flow of the calling process performed in the communication system which concerns on 1st Embodiment. It is a figure which shows an example of a call-out screen and a call-out button. It is a figure which shows an example of a voice call screen and a cutting | disconnection button. It is a sequence diagram which shows an example of the flow of the calling process performed in the communication system which concerns on 1st Embodiment. It is a sequence diagram which shows an example of the flow of the incoming call process performed in the communication system which concerns on 1st Embodiment. It is a figure which shows an example of a call-in screen and a call-in button. It is a sequence diagram which shows an example of the flow of the incoming call process performed in the communication system which concerns on 1st Embodiment. It is a figure which shows the whole figure of the message sequence performed in the communication system which concerns on 1st Embodiment. FIG. 10 shows a detailed RTSP message sequence of a capability negotiation procedure. FIG. 7 shows a detailed RTSP message sequence of the session establishment procedure. It is a figure which shows the RTSP message sequence for performing PAUSE setting of content transmission / reception. It is a figure which shows the RTSP message sequence for performing PAUSE cancellation | release setting of content transmission / reception. FIG. 5 is a diagram illustrating a UIBC RTSP message sequence. It is a sequence diagram which shows an example of the flow of the calling process performed in the communication system which concerns on 2nd Embodiment. It is a sequence diagram which shows an example of the flow of the calling process performed in the communication system which concerns on 2nd Embodiment. It is a sequence diagram which shows an example of the flow of the incoming call process performed in the communication system which concerns on 2nd Embodiment. It is a sequence diagram which shows an example of the flow of the incoming call process performed in the communication system which concerns on 2nd Embodiment. It is a figure which shows the example of a display of the environment setting screen which concerns on 3rd Embodiment. It is a figure which shows the example of a display of the environment setting screen which concerns on 3rd Embodiment. It is a block diagram showing an example of rough composition of a smart phone. It is a block diagram which shows an example of a schematic structure of a car navigation apparatus.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration will be assigned the same reference numerals and redundant description will be omitted.

The description will be made in the following order.
1. System overview 2. Configuration example of sink device 3. Configuration Example of Source Device 4. Operation processing 4-1. Basic operation 4-2. Specific example of operation processing 4-2-1. Operation when incoming / outgoing of sink device 4-2-2. Operation when incoming / outgoing of source device 4-3. Flow of message transmission and reception 4-4. Supplement 5. Second embodiment 6. Third Embodiment Application example 7-1. First application example 7-2. Second application example 8. Summary

<1. System Overview>
FIG. 1 is an explanatory diagram for describing an overview of a communication system (information processing system) 1 according to an embodiment. The communication system 1 includes a sink device (information processing device) 100 and a source device (information processing device) 200. The sink device 100 establishes a wireless connection with the source device 200. Also, the sink device 100 can establish a wireless connection with the external device 400 via the source device 200 and the base station 300.

  The sink device is a receiving device that reproduces content received from the source device. In the example of FIG. 1, the sink device 100 is an in-vehicle monitor. The on-vehicle monitor 100 has, for example, a microphone 151 and a speaker 152 for voice input and output. The source device is a transmitting device that transmits content to the sink device. In the communication system 1, the content transmitted from the source device to the sink device typically includes at least one of video content and audio content. In the example of FIG. 1, the source device 200 is a smartphone. Further, in the example of FIG. 1, the external device 400 is a smartphone. The technology according to the present disclosure is not limited to the illustrated example, and, for example, a television apparatus, a projector, a digital still camera, a digital video camera, a tablet PC (Personal Computer), a desktop PC, a laptop PC, a PDA The present invention is applicable to any type of sink device and source device, such as Personal Digital Assistant), mobile phones, gaming devices, wearable devices or storage devices.

  The wireless connection between the sink device 100 and the source device 200 is, for example, a wireless local area network (LAN) system such as IEEE (Institute of Electrical and Electronics Engineers) 802.11a, 11b, 11g, 11n, 11ac or 11ad. It may be formed according to any kind of wireless communication method such as wireless PAN (Personal Area Network) method such as UWB (Ultra Wideband) or Zigbee (registered trademark), or wireless MAN (Metropolitan Area Network) method such as IEEE 802.16. . In addition, an intermediate device such as a wireless access point may or may not be interposed between the sink device 100 and the source device 200. Also, the source device 200 and the external device 400 are wirelessly connected using, for example, a public network wireless communication network (for example, a third generation mobile communication system) via the base station 300.

  In the following description, as an example, the sink device 100 and the source device 200 are connected by a wireless LAN method, and content is transmitted according to the messaging specification of the Wi-Fi display over the wireless LAN connection. For example, Miracast can be used as the short distance wireless AV (Audio Visual) transmission communication used for this wireless LAN connection. In Miracast, wireless connection is established between the sink device 100 and the source device 200 using Wi-Fi Direct or a connection protocol called TDLS (Tunneled Direct Link Setup). Then, the audio and the display image reproduced by the source device 200 can be transmitted to the sink device 100, and the audio and image data can be similarly output (mirrored) by the sink device 100 as well. Furthermore, a control channel for exchanging user input information may be formed between the sink device 100 and the source device 200. The control channel may be, for example, a Transmission Control Protocol / Internet Protocol (TCP / IP) -based User Input Back Channel (UIBC). Note that, instead of the Wi-Fi display, the content may be transmitted using another remote desktop application such as, for example, VNC (Virtual Network Computing).

  Also, the sink device 100 and the source device 200 have a dual-role function that enables not only unidirectional content transmission from the source device to the sink device but also bidirectional content transmission from the sink device to the source device. It may be According to this function, the sink device 100 and the source device 200 can perform Miracast connection bi-directionally, and can respectively exhibit both the function as the sink device and the function as the source device. That is, the sink device 100 can not only receive and reproduce the content but also transmit and reproduce the content to another wireless device. Similarly, the source device 200 may not only transmit the content to another wireless device for reproduction but also receive and reproduce the content. For example, in the example illustrated in FIG. 1, it is assumed that the smartphone 200 and the smartphone 400 perform voice communication such as telephone calls. For the sound from the smartphone 400, the smartphone 200 functions as a source device, and the in-vehicle monitor 100 functions as a sink device. Specifically, the smartphone 200 transmits the voice received from the smartphone 400 to the on-vehicle monitor 100, and the on-vehicle monitor 100 outputs the voice of the other party from the speaker 152. On the other hand, for the voice of the user input to the microphone 151, the in-vehicle monitor functions as a source device, and the smartphone 200 functions as a sink device. Specifically, the in-vehicle monitor 100 transmits the user voice input to the microphone 151 to the smartphone 200, and the smartphone 200 transmits the received user voice to the smartphone 400. Accordingly, the communication system 1 enables voice communication between the smartphone 200 and the smartphone 400 using the microphone 151 and the speaker 152. As described above, although the sink device 100 and the source device 200 can have both the function as the sink device and the function as the source device, the sink device 100 can be used as the sink device or the source unless otherwise specified. The device 200 will be described as a source device.

  As described above, in Miracast, technology relating to playback control of content is mainly defined, and how the source device and the sink device work in cooperation with each other for user operation is under development. For example, in the example illustrated in FIG. 1, the screen displayed on the smartphone 200 is mirrored on the in-vehicle monitor 100 by using Miracast, and the same screen is displayed on both touch panels. In the example shown in FIG. 1, “incoming call” button “outgoing call” button is displayed on both sides. The user can, for example, answer a call received from the external device 400 by touching the “incoming call” button. Also, the user can make a call, for example, to the external device 400 by touching the “call origination” button. However, even if the user touches the button on the in-vehicle monitor 100, the in-vehicle monitor 100 can not uniquely recognize and process this user operation. For example, when the user touches the "Incoming call" button on the in-vehicle monitor 100, the in-vehicle monitor 100 once transmits this operation information to the smartphone 200, and starts the microphone 151 and the speaker 152 according to the returned instruction. Etc. were processed. Furthermore, even when trying to uniquely recognize a user operation on the in-vehicle monitor 100 side, it is difficult to determine whether the user operation is performed on the in-vehicle monitor 100 or the smartphone 200. Therefore, for example, even if an instruction to increase the volume is given on the mirrored screen from the smartphone 200, the in-vehicle monitor 100 determines which of the smartphone 200 or the in-vehicle monitor 100 should increase the volume of. It was difficult. Therefore, each embodiment in which the sink device can directly execute the processing according to the user operation without requiring recognition of the user operation by the source device and an instruction to the sink device will be described in detail from the next section. Explain to.

<2. Configuration Example of Sink Device>
FIG. 2 is a block diagram showing an example of the configuration of the sink device 100 according to an embodiment. Referring to FIG. 2, the sink device 100 includes a wireless communication unit 110, a stream transmission / reception unit 120, a compression / decompression unit 130, a display unit 140, an audio input / output unit 150, a control signal transmission / reception unit 160, a control unit 170, and a user interface unit. It has 180. As described above, although the sink device 100 may also have a function as a source device, the function as a source device will be described in detail later with reference to a configuration example of the source device 200, so Detailed description is omitted.

(1) Wireless communication unit 110
The wireless communication unit 110 is a wireless interface that mediates wireless communication with another device by the sink device 100. In the present embodiment, the wireless communication unit 110 establishes a wireless connection with the source device 200. Then, the wireless communication unit 110 receives, via an antenna, a wireless signal including content data transmitted by the source device 200 over a wireless connection. The wireless communication unit 110 outputs a reception signal including content data to the stream transmission / reception unit 120. The wireless communication unit 110 can also transmit and receive control messages (control information) to and from the source device 200. The control message transmitted to the source device 200 is generated by the control unit 170 described later. Further, the control message received from the source device 200 is interpreted by the control unit 170.

  The wireless communication unit 110 may be able to use a plurality of frequency channels different in transmission rate in parallel or selectively. The plurality of frequency channels may be channels having transmission rates of 2.4 GHz, 5 GHz and 60 GHz, for example. The wireless communication unit 110 may switch the frequency channel to be used for receiving the content from the source device 200 according to the assignment by the control unit 170. The wireless communication unit 110 may use frequency channel information and change the frequency to be used. Also, the wireless communication unit 110 measures the connection quality of each frequency channel, and outputs a quality indicator (for example, received signal strength or SNR (Signal-to-Noise Ratio) etc.) indicating the measurement result to the control unit 170. It is also good. Of course, other frequency channels other than 2.4 GHz, 5 GHz and 60 GHz may be used.

(2) Stream transceiver 120
The stream transmission / reception unit 120 acquires a bit stream of content to be reproduced (for example, video content or audio content) from the reception signal input from the wireless communication unit 110. Then, the stream transmission / reception unit 120 outputs the bit stream acquired for the content to be reproduced to the compression / decompression unit 130.

The stream transmission / reception unit 120 measures a quality indicator (for example, BER (Bit Error Rate) or PER (Packet Error Rate), etc.) for evaluating connection quality of the wireless connection for each stream, and sends the quality indicator to the control unit 170. You may output it.

(3) Compression / decompression unit 130
The compression and decompression unit 130 decodes each content from the bit stream of the content received from the source device 200. The compression / decompression unit 130 outputs the decrypted video content to the display unit 140 when the video content is decrypted. When the audio expansion unit 130 decodes audio content, the compression / decompression unit 130 outputs the decoded audio content to the audio input / output unit 150. The compression and decompression unit 130 can decode a plurality of contents in parallel. When the content is compressed and encoded, the content is expanded through the decoding process in the compression and expansion unit 130. The compression / decompression unit 130 may realize the decoding process by software, or may realize the decoding process by hardware.

(4) Display unit 140
The display unit 140 and the audio input / output unit 150 are used to reproduce the content decoded by the compression / decompression unit 130. The display unit 140 displays an image based on the image data received from the source device 200 by the wireless communication unit 110 and decoded and expanded by the compression and expansion unit 130. For example, the display unit 140 sequentially outputs each frame of the decoded video content. The display unit 140 can adjust a display attribute (for example, a frame size or the like) of the video content to be reproduced in accordance with the desired window configuration and the specification of the display of the output destination. Thus, even if the screen sizes of the sink device 100 and the source device 200 are different, for example, the display unit 140 may output a screen of an appropriate size by expanding and contracting the image mirrored from the source device 200. . Also, the display unit 140 may be plural, and for example, the control unit 170 may change the display unit 140 to output according to the resolution of each display unit 140, the number of the display units 140, and the like. The display unit 140 is realized by, for example, a display panel such as an organic EL (Electro-Luminescence) panel or an LCD (Liquid Crystal Display) panel. In addition, the display unit 140 may be realized as a touch panel capable of performing an operation input by the user touching or approaching a display surface with a finger or the like.

(5) Voice input / output unit 150
The voice input / output unit 150 has a function of inputting and outputting voice, and is realized by, for example, the speaker 152 and the microphone 151 shown in FIG. The audio input / output unit 150 sequentially outputs the audio signal of the audio content decoded and expanded by the compression and expansion unit 130. A plurality of audio input / output units 150 may be provided. For example, the control unit 170 may output the audio input / output units 150 according to the audio reproduction capability of each audio input / output unit 150, the number of audio input / output units 150 You may change it.

(6) Control signal transmission / reception unit 160
The control signal transmission / reception unit 160 performs transmission processing for transmitting a control signal (control message) output from a control unit 170 described later to another wireless communication apparatus such as the source device 200 via the wireless communication unit 110. Further, the control signal transmission / reception unit 160 outputs, to the control unit 170, a control message received by the wireless communication unit 110 from another wireless communication device such as the source device 200, for example.

(7) Control unit 170
The control unit 170 functions as an arithmetic processing unit and a control unit, and controls the overall operation in the sink device 100 according to various programs. For example, the control unit 170 negotiates the image content and the audio content with the source device 200 according to various setting information input by the user interface unit 180 described later. Specifically, the control unit 170 is based on the control message received by the control signal transmission / reception unit 160, the performance of each device included in the sink device 100, and various setting information input by the user interface unit 180 described later. Control the operation of the source device 200 that is the source of the content.

For example, the control unit 170 controls the stream transmitted from the source device 200 so as not to exceed the processing capability of the sink device 100 itself. Specifically, based on the request decoding capability required by the display unit 140 and the content reproduced by the audio input / output unit 150 and the actual decoding capability of the compression and expansion unit 130, the control unit 170 determines the source of the content transmission source. A control message may be generated to control the coding condition of content in the device 200. In addition, the control unit 170 may generate a control message for performing band control of the content stream transmitted from the source device 200 of the transmission source of the content based on the setting information on the band. Further, the control unit 170 changes the resolution of the image displayed on the display unit 140 based on the setting information on the resolution of the display, and sends a control message for requesting the source device 200 to have a transmission rate equivalent to this resolution. It may be generated. Further, for example, the control unit 1 70 based on the setting information about the screen size of the display may generate a control signal for changing the display area and the size of the image displayed on the display unit 140. These generated control messages are transmitted to the source device 200 via the control signal transmission / reception unit 160 and reflected in processing in the source device 200.

  In addition, the display unit 140 may notify, for example, the user of the content of the change by the control message. The control unit 170 may also have a function related to block control of the entire communication system 1. For example, the control unit 170 may have a function of setting the frequency channel to be used and the resolution based on the information from the stream transmission / reception unit 120 and the user interface unit 180.

  As described above, the control unit 170 generates a control message for negotiation, and transmits the control message to the source device 200 via the control signal transmission / reception unit 160. On the other hand, the control unit 170 controls each device included in the sink device 100 based on the control message received from the source device 200.

  The control message received from the source device 200 may store various information. For example, among the images displayed on the display unit 140, the control message includes information specifying an area where the user interface unit 180 should acquire an operation, information indicating the type of operation to be acquired, and an operation on the area It includes a combination of information indicating processing to be controlled by the control unit 170 when it is received. The first of the combination information is also referred to as coordinate information, the second is also referred to as type information, and the third is also referred to as command information. Further, the area specified by the coordinate information is also referred to as a command area. The command area specified by the coordinate information may have, for example, a square, a rounded square, a triangle, a circle, and various other shapes. Examples of the type of operation indicated by the type information include various operations such as one-point touch, multi-point touch, swipe, pinch in, and pinch out. The control unit 170 controls to perform processing indicated by command information when a user operation matching the type information is acquired in the command area. Examples of the processing indicated by the command information include various processing such as setting of hardware or software of the sink device 100, ON / OFF processing, etc., regarding call origination, incoming call, mail transmission / reception, Internet browsing. The control unit 170 controls each device in the sink device 100 to perform processing in accordance with command information in accordance with a user operation on the command area acquired by the user interface unit 180. Note that control of each device in the sink device 100 so that the control unit 170 performs processing according to command information is hereinafter also referred to simply as executing a command. Further, a combination of coordinate information, type information, and command information is also referred to as operation setting information, and a control message including the operation setting information is hereinafter also referred to as an operation setting message. A specific example of the command area and the command information will be specifically described with reference to FIG.

FIG. 3 is a diagram for explaining a command area and command information. The left drawing of FIG. 3 shows the screen of the source device 200, and the right drawing of FIG. 3 shows the screen of the sink device 100. The screen of the sink device 100 is displayed on the touch panel. As shown in FIG. 3, the screen of the source device 200 is mirrored to the sink device 100, and an incoming call button is displayed. Here, an operation setting message including a combination of coordinate information of an incoming call button, type information indicating a touch operation, and command information indicating processing to be performed when the incoming call button is touched is received from the source device 200. Think about the case. The coordinate information specifies the area of the incoming call button which is the command area, for example, by upper left coordinates (X1, Y1 ) and lower right coordinates (X2, Y2). Based on this operation setting message, when the call reception button is operated, the control unit 170 turns on the microphone 151, for example, to validate the voice input by the voice input / output unit 150. Then, the sink device 100 can transmit the microphone sound acquired by the microphone 151 to the voice call partner by exhibiting the function as a source device described later.

  The operation setting message may include flag information indicating either valid or invalid of the operation setting information. The control unit 170 executes the command when the command area related to the valid operation setting information is touched. On the other hand, when the control unit 170 is notified by the source device 200 that the operation setting information is invalidated, the control unit 170 does not execute the command even if the command area related to the invalid operation setting information is touched. That is, the control unit 170 monitors the user operation to the command area from the notification of the validity of the operation setting information to the notification of the invalidity, and when the designated user operation is acquired, the control unit 170 Run. Hereinafter, among the operation setting messages, a flag information is valid, that is, a message for validating the operation setting information is also referred to as a validation message. Also, a message in which the flag information is invalid, that is, the message for invalidating the operation setting information is also referred to as an invalidation message. Hereinafter, an example of the operation setting message regarding the incoming call button shown in FIG. 3 will be shown.

<Activation message>
Coordinate information: (X1, Y1) = (10, 300), (X2, Y2) = (100, 400)
Type information: (touch)
Command information: (Incoming call)
Flag information: (valid)

<Invalidation message>
Coordinate information:
(X1, Y1) = (10, 300), (X2, Y2) = (100, 400)
Type information: (touch)
Command information: (Incoming call)
Flag information: (disabled)

  The control unit 170 monitors the area specified by the coordinates (10, 300) and the coordinates (100, 400) as a command area from the reception of the activation message to the reception of the invalidation message as a command area. Execute command related to incoming call when it is done. The operation setting message may include a plurality of operation setting information, or valid operation setting information and invalid operation setting information may be mixed. The control unit 170 also has a function of converting the coordinates of the command area according to the difference in resolution, screen size, display direction, etc. of the display. Therefore, the coordinates of the command area in the sink device 100 and the coordinates of the incoming call button in the source device 200 may be the same or different.

  Here, depending on the command to be executed, it may not be possible with the sink device 100 alone, and processing linked by the source device 200 may also be required. In this case, the control unit 170 transmits a message for causing the source device 200 to execute a process corresponding to the process indicated by the command information to the source device 200 via the wireless communication unit 110. This message is also referred to below as a collaboration request message. The cooperation request message may include information instructing a process to be performed by the source device 200. Also, the cooperation request message may simply include operation information indicating the user operation acquired by the user interface unit 180, or the source device 200 may determine the processing to be executed based on the operation information. For example, the control unit 170 controls the control signal transmission / reception unit 160 to transmit such a cooperation request message using TCP / IP and UIBC. When the incoming call button is touched, the control unit 170 controls the control signal transmission / reception unit 160 to transmit a cooperation request message including operation information on the UIBC, and the source device 200 performs a voice call based on the operation information. Process the incoming call of For example, in the case of an incoming call, the source device 200 establishes communication with the external device 400 of the caller. As a result, while communication is established by the source device 200, voice communication can be performed by the sink device 100. Here, although the incoming call has been described with reference to FIG. 3, the same applies to, for example, an outgoing call. Also, as another example of the cooperation request message, the control unit 170 may transmit a message for PAUSE setting or PLAY setting for RTSP. For example, after establishing the Miracast connection, the control unit 170 performs PAUSE setting while transmission and reception of content is not performed, and performs PAUSE cancellation setting (PLAY setting) while transmission and reception of content is performed. Communication can be avoided.

  In addition, the control unit 170 may transmit and receive control messages including commands related to content reproduction (for example, reproduction start, reproduction stop, fast forward and rewind, etc.) via a control channel between devices. . The control unit 170 can control the operation of the display unit 140 and the voice input / output unit 150 in response to the detection of such a command.

(8) User interface unit 180
The user interface unit 180 receives user input via an input device (not shown) of the sink device 100. The user interface unit 180 may receive, for example, an input signal generated in unique hardware such as a remote controller, a touch panel, a keyboard, a mouse, a button, or a switch as a user input. Also, the user interface unit 180 may receive, as user input, a voice command acquired through a microphone, a gesture command acquired through a camera, or a sensor command acquired through various sensors such as a gyro sensor and a human sensor. . The user interface unit 180 according to the present embodiment is an acquisition unit that acquires an operation on a command area indicated by the operation setting message received from the source device 200 by the wireless communication unit 110 among the images displayed by the display unit 140. Function. In the present specification, the user interface unit 180 is described as being realized by a touch panel integrally formed with the display unit 140. The user interface unit 180 outputs the received user input to the control unit 170.

  The configuration example of the sink device 100 has been described above. Subsequently, a configuration example of the source device 200 will be described with reference to FIG.

<3. Configuration Example of Source Device>
FIG. 4 is a block diagram showing an example of the configuration of the source device 200 according to an embodiment. Referring to FIG. 4, the source device 200 includes a wireless communication unit 210, a stream transmission / reception unit 220, a compression / decompression unit 230, a content acquisition unit 240, a content reproduction unit 250, a control signal transmission / reception unit 260, a control unit 270, and a user interface unit. It has 280. As described above, although the source device 200 may also have a function as a sink device, the function as the sink device has been described in detail in the configuration example of the sink device 100, and thus the source device 200 may be a sink device here. Detailed description of the functions is omitted. The wireless communication unit 210 of the source device 200 may be regarded as a first wireless communication unit, and the wireless communication unit 110 of the sink device 100 may be regarded as a second wireless communication unit. Also, it is possible to grasp the control unit 270 of the source device 200 as a first control unit, and the control unit 170 of the sink device 100 as a second control unit.

(1) Wireless communication unit 210
The wireless communication unit 210 is a wireless interface that mediates wireless communication between the source device 200 and another device. In the present embodiment, the wireless communication unit 210 establishes a wireless connection with the sink device 100. The wireless communication unit 210 transmits a wireless signal including content data generated by the stream transmission / reception unit 220 to the sink device 100 over a wireless connection via an antenna. The wireless communication unit 210 can also transmit and receive a wireless signal including a control message to and from the sink device 100. The control message transmitted to the sink device 100 is generated by the control unit 270 described later. The control message received from the sink device 100 is interpreted by the control unit 270.

  Also, the wireless communication unit 210 establishes a wireless connection with the external device 400 using a public network wireless communication network (for example, a third generation mobile communication system). The wireless communication unit 210 transmits a wireless signal including the content data generated by the stream transmission / reception unit 220 to the external device 400 over a wireless connection via an antenna. Also, the wireless communication unit 210 receives a wireless signal including content data from the external device 400 over a wireless connection via an antenna.

(2) Stream transceiver 220
The stream transmission / reception unit 220 transmits the bit stream of encoded content input from the compression / decompression unit 230 described later to the sink device 100 via the wireless communication unit 210. The stream transmission / reception unit 220 may generate a multimedia stream by multiplexing the stream of video content and the stream of audio content, and may transmit the generated multimedia stream.

(3) Compression / decompression unit 230
The compression and decompression unit 230 encodes the content to be transmitted to the sink device 100, which is input from the content acquisition unit 240 described later, and generates a bit stream of the content. Then, the compression / decompression unit 230 outputs the generated bit stream to the stream transmission / reception unit 220. The coding condition of the content in the compression and expansion unit 230 is controlled by the control unit 270 based on the control message received from the sink device 100. For example, the coding conditions for video content may include codec type, resolution, frame rate and quality level of the video codec used. The coding conditions for audio content may include the codec type, sampling rate and sound quality level of the audio codec used. When instructed not to compress and encode the content, the compression and decompression unit 230 may generate a bit stream of the content in an uncompressed format without compressing the content.

  The compression and decompression unit 230 may have one or more coding circuits as hardware. In addition, the compression and decompression unit 230 may have a processor capable of executing software modules for video coding and audio coding. The compression and decompression unit 230 may have both a coding circuit as hardware and a processor capable of executing a software module. The compression and decompression unit 230 may support any codec type.

(4) Content acquisition unit 240
The content acquisition unit 240 acquires content to be provided to the sink device 100 from a storage unit (not shown) or another data source, and outputs the acquired content to the compression and decompression unit 230. The other data source is, for example, wirelessly connected to the camera and the microphone of the source device 200 or a remote device accessible by the source device 200 (for example, a content server or a web camera with a microphone) or the source device 200 External device 400 may be included. Which content should be provided to the sink device 100 may be specified by a user input detected by the user interface unit 280 or by a control message received from the sink device 100. When the acquired content is encoded by a codec type different from the encoding condition instructed from the sink device 100, the content acquisition unit 240 temporarily decodes the content and then outputs the content to the compression / decompression unit 230. It is also good.

(5) Content reproduction unit 250
The content reproduction unit 250 has a function of reproducing the content received from the sink device 100 or the external device 400 via the wireless communication unit 210 and decoded by the compression / decompression unit 230. For example, the content reproduction unit 250 can be realized by a display panel such as an organic EL panel or an LCD panel. In addition, the content reproduction unit 250 may be realized as a touch panel capable of performing an operation input by a user touching or approaching a display surface with a finger or the like. Besides, the content reproduction unit 250 can be realized by, for example, a speaker. The audio and the image may be output from both the sink device 100 and the source device 200. However, it is desirable that voice be output from only one of the sink device 100 and the source device 200 by negotiation using a control message.

(6) Control signal transmission / reception unit 260
The control signal transmission / reception unit 260 performs transmission processing of transmitting a control message output from the control unit 270 described later to another wireless communication apparatus such as the sink device 100 via the wireless communication unit 210. Further, the control signal transmission / reception unit 260 outputs, to the control unit 270, a control message received by the wireless communication unit 210 from another wireless communication device such as the sink device 100.

(7) Control unit 270
The control unit 270 functions as an arithmetic processing unit and a control unit, and controls the overall operation in the source device 200 according to various programs. For example, the control unit 270 negotiates image content and audio content with the sink device 100, and controls each device in the source device 200 to operate based on the negotiation result.

  For example, based on the control message received from the sink device 100, the control unit 270 controls the content encoding condition in the compression and decompression unit 230. In addition, the control unit 270 may perform control for changing the resolution of the image data to be transmitted or control for changing the image area of the image data to be transmitted based on the control message. Further, the control unit 270 may perform bandwidth control of the media stream to be transmitted based on the control message. For this band control, the control unit 270 may measure a quality indicator (for example, BER or PER) for evaluating the connection quality of the wireless connection with the sink device 100.

  Also, the control unit 270 may control the entire communication system 1 according to the negotiation result with the sink device 100. For example, the control unit 270 may further control the frequency channel of the wireless connection with the sink device 100. For example, the control unit 270 obtains the power state of each device in the communication system 1 (for example, battery-powered or connected to a power source, or the remaining amount of battery, etc.) through message exchange with the sink device 100. Do. Then, the control unit 270 may select a frequency channel to be used, for example, in order to suppress battery consumption in the battery-powered source device 200. Also, if it is recognized that the bit rate of the content to be transmitted to the sink device 100 is high, the wireless connection with the sink device 100 has a higher transmission rate (or a better connection quality). Frequency channels may be assigned.

  The control unit 270 may also control the operation of the sink device 100 by exchanging messages with the sink device 100. Examples of the message transmitted to the sink device 100 include an operation setting message. The source device 200 monitors screen transition and the like of the screen displayed on the sink device 100, transmits an activation message when the operable area is displayed, and is invalidated when the area is hidden. Send an authentication message. As the operable area, for example, in addition to the incoming call button shown in FIG. 3, an outgoing call button, a mail transmission / reception button, various operation buttons of the Internet browser, etc. may be mentioned. Of course, in addition to the button, an area in which an arbitrary operation such as a swipe operation can be performed may be included in the operable area.

  For example, first, the control unit 270 monitors the content acquisition unit 240 and determines whether an image displayed on the sink device 100 via the content transmission by the wireless communication unit 210 includes an operable area. Do. When the operable area is included, the control unit 270 transmits an operation setting message related to the area to the sink device 100. Specifically, the control unit 270 indicates coordinate information for specifying this area, type information indicating the type of operation to be acquired, and a command indicating processing to be executed by the sink device 100 when acquiring an operation to this area. Generate an activation message that contains action configuration information that consists of a combination of information. Then, the control unit 270 transmits the generated validation message to the sink device 100 via the wireless communication unit 210. On the other hand, the control unit 270 monitors the content acquisition unit 240 and, from the image displayed on the sink device 100 via the content transmission by the wireless communication unit 210, does not include an area operable by, for example, screen transition. It is determined whether or not. When the operable area is not included, the control unit 270 generates an invalidation message including operation setting information related to the area. Then, the control unit 270 transmits the generated invalidation message to the sink device 100 via the wireless communication unit 210. Note that the wireless communication unit 210 may transmit this operation setting message using an extension of one of the messages defined in Miracast, or transmits it using another message not defined. It is also good.

  In addition, the message received from the sink device 100 may include the above-described cooperation request message that requests a process corresponding to a command executed in the sink device 100. When the cooperation request message is received via the wireless communication unit 210, the control unit 270 controls each device in the source device 200 to execute the process indicated by the received cooperation request message. In the example shown in FIG. 3, when the incoming call button is touched in the sink device 100, the control unit 270 controls each device in the source device 200 to perform processing corresponding to the incoming call. For example, the control unit 270 controls the wireless communication unit 210 to establish a wireless connection with the external device 400 based on a cooperation request message including operation information. In addition, the control unit 270 may perform PAUSE setting or PAUSE cancellation setting (PLAY setting) of transmission / reception of content based on a message for PAUSE setting or PLAY setting of RTSP.

  Furthermore, when the user input related to the reproduction of the content is detected in the user interface unit 280, the control unit 270 transmits a control message including a command corresponding to the detected user input to the control channel between the devices. It may be transmitted to the sink device 100. For example, when the source device 200 is selected as a standby device, the wireless communication unit 210 can receive a control message instructing stop of content reproduction. In this case, the control unit 270 may shift the operation mode of the source device 200 to the standby mode, and partially stop the power supply to the respective units of the source device 200 during the standby mode. In the standby mode, the wireless communication unit 210 may operate, for example, intermittently, and receive a control message from the sink device 100 in a periodic active period. That is, the wireless connection between the sink device 100 and the source device 200 may be maintained during the standby mode. The control unit 270 may return the operation mode of the source device 200 to the active mode when a control message instructing start of content reproduction is received during the standby mode.

(8) User interface unit 280
The user interface unit 280 receives user input via an input device (not shown) of the source device 200. The user interface unit 280 may receive, for example, an input signal generated in unique hardware such as a remote controller, a touch panel, a keyboard, a mouse, a button, or a switch as a user input. In addition, the user interface unit 280 may receive, as user input, a voice command acquired through a microphone, a gesture command acquired through a camera, or a sensor command acquired through various sensors such as a gyro sensor and a human sensor. . In the present specification, the user interface unit 280 is described as being realized by a touch panel integrally formed with the display. The user interface unit 280 outputs the received user input to the control unit 270.

  The configuration example of the source device 200 has been described above. Subsequently, an operation process by the communication system 1 will be described.

<4. Operation processing>
[4-1. basic action]
First, basic operation processing of the sink device 100 and the source device 200 will be described with reference to FIGS. 5 and 6.

(Basic Operation of Sink Device 100)
FIG. 5 is a diagram showing an example of the flow of basic operation processing executed by the sink device 100 according to the present embodiment. Referring to FIG. 5, first, the control unit 170 of the sink device 100 determines whether or not the activation message from the source device 200 has been received by the control signal transmission / reception unit 160 (step S10). When it is determined that it has not been received (step S10 / NO), the process returns to step S10 again, and the control unit 170 repeats the reception confirmation of the validation message. On the other hand, when it is determined that the validation message has been received (step S10 / YES), the control unit 170 validates the command area (S12). Specifically, the control unit 170 starts monitoring the user operation on the command area. Next, the control unit 170 determines whether the invalidation message related to the command area from the source device 200 is received by the control signal transmission / reception unit 160 (step S14). If it is determined that it has not been received (step S14 / NO), the process returns to step S14 again, and the control unit 170 continues monitoring of the command area. When a user operation matching the type information to the command area is acquired by the user interface unit 180 during monitoring, the control unit 170 executes the command specified in the validation message, and executes the cooperation request message to the source device 200. Send to On the other hand, when it is determined that the invalidation message has been received (step S14 / YES), the control unit 170 invalidates the command area (step S16). Specifically, the control unit 170 stops monitoring of the user operation on the command area. Note that this operation processing can also process a plurality of messages.

(Basic operation of source device 200)
FIG. 6 is a diagram showing an example of the flow of basic operation processing executed by the source device 200 according to the present embodiment. Referring to FIG. 6, first, the control unit 270 of the source device 200 monitors screen transition and the like of the screen displayed on the sink device 100, and determines whether or not an operable area is displayed (step S20). If it is determined not to be displayed (step S20 / NO), the process returns to step S20 again, and the control unit 270 continues monitoring of the screen displayed on the sink device 100. On the other hand, when it is determined to be displayed (step S20 / YES), control unit 270 generates an activation message including operation setting information for designating the area as a command area, and via wireless communication unit 210. It transmits to the sink device 100 (step S22). Next, the control unit 270 monitors screen transition and the like of the screen displayed on the sink device 100, and determines whether the displayed operable area is not displayed (step S24). If it is determined that the display is continued (step S24 / NO), the process returns to step S24 again, and the control unit 270 continues monitoring of the screen displayed on the sink device 100. On the other hand, when it is determined that the display is not displayed (step S24 / YES), the control unit 270 generates an invalidation message related to the area, and transmits it to the sink device 100 via the wireless communication unit 210 (step S26). ). Note that this operation processing can also process a plurality of messages.

[4-2. Specific example of operation processing]
Subsequently, specific operation processing of the sink device 100 and the source device 200 will be described with reference to FIGS. 7 to 10. Here, as an example, a flow of incoming call processing and outgoing call processing performed in the sink device 100 and the source device 200, which uses a touch operation performed by the sink device 100 as a trigger, will be described.

[4-2-1. Operation when incoming / outgoing from sink device 100]
7 and 8 are diagrams showing an example of the flow of incoming call processing and outgoing call processing performed by the sink device 100 according to the present embodiment.

  Referring to FIG. 7, first, the control unit 170 of the sink device 100 performs initialization processing and preparation processing of each device in the sink device 100 (step S102). Next, the control unit 170 controls the wireless communication unit 110 to establish a wireless connection with the wireless communication unit 210 of the source device 200 (step S104). Next, the wireless communication unit 110 bidirectionally establishes a Miracast connection with the source device 200 (step S106). Specifically, first, the control unit 170 controls the control signal transmission / reception unit 160 to exchange various control messages with the control signal transmission / reception unit 260 of the source device 200 to perform various negotiations. Then, the control unit 170 controls the stream transmission / reception unit 120 and the compression / decompression unit 130 to decode the content data from the wireless signal received from the source device 200 by the wireless communication unit 110, the display unit 140 and the voice input / output unit Output from 150. By this processing, the sink device 100 can operate as a sink device to reproduce the content transmitted from the source device. Further, the control unit 170 controls the compression / decompression unit 130 and the stream transmission / reception unit 120 to convert the audio data acquired by the audio input / output unit 150 into a wireless signal, and transmits the source device 200 via the wireless communication unit 110. Send to By this processing, the sink device 100 can operate as a source device and can reproduce the content transmitted to the source device 200 functioning as the sink device. With such a two-way connection, for example, when making a voice call such as a telephone, it is possible to reproduce the voice of the other party by the speaker 152 and transmit the user voice acquired by the microphone 151 to the other party. Here, since transmission of the microphone sound from the sink device 100 to the source device 200 is unnecessary while the voice call is not being performed, the control unit 170 performs PAUSE setting of Miracast to transmit the sound to the source device 200. It is preferable to suspend transmission of data. Note that the PAUSE message may be implemented using the Real Time Streaming Protocol (RTSP) M9 message.

  Next, the control unit 170 determines whether the activation message of the incoming call button has been received by the control signal transmission / reception unit 160 (step S108). If it is determined that it is not received (step S108 / NO), the control unit 170 determines whether or not the activation signal of the calling button has been received by the control signal transmitting / receiving unit 160 (step S126). When it is determined that none of them has been received (step S126 / NO), the process returns to step S108 again.

  If it is determined that the activation message of the incoming call button has been received (step S108 / YES), the control unit 170 activates the incoming call button (step S110). For example, the control unit 170 starts monitoring whether the touch to the command area of the incoming call button specified by the coordinate information included in the activation message of the incoming call button is acquired by the user interface unit 180. .

  When the incoming call button is not touched (NO in step S112), the control unit 170 determines whether the invalidation message of the incoming call button is received by the control signal transmitting / receiving unit 160 (step S128). If it is determined that the call has not been received (step S128 / NO), the process returns to step S112 again, and the control unit 170 continues monitoring of the touch on the incoming call button. On the other hand, when it is determined that it has been received (step S128 / YES), the control unit 170 invalidates the incoming call button (step S130). Specifically, the control unit 170 stops monitoring whether the touch to the command area of the incoming call button specified by the coordinate information included in the invalidation message of the incoming call button is acquired by the user interface unit 180. Do. Then, the process returns to step S108 again.

  When the incoming call button is touched (step S112 / YES), the control unit 170 executes the incoming call button command indicated by the command information included in the incoming call button activation message (step S114). For example, the control unit 170 turns on the microphone 151 to enable voice input. Further, the control unit 170 performs PAUSE release setting (PLAY setting) of Miracast, and resumes transmission of audio data to the source device 200. In addition, the control unit 170 controls the control signal transmission / reception unit 160 to transmit a cooperation request message related to the incoming call button to the source device 200. For example, the control unit 170 transmits, as a cooperation request message, a message including operation information indicating that the incoming call button has been touched, and a PLAY setting message. The PLAY setting message may be implemented using the RTSP M7 message. Thereby, for example, a wireless connection between the source device 200 and the called party is established, and a voice call is started. Therefore, for example, the screen changes from the incoming call screen to the voice call screen, the incoming call button is not displayed, and the disconnection button is displayed. Along with this, the source device 200 transmits the invalidation message of the incoming call button and the activation message of the disconnect button to the sink device 100. When the invalidation message of the incoming call button is received by the control signal transmission / reception unit 160, the control unit 170 invalidates the incoming call button (step S116).

  In addition, when the activation message of the disconnection button is received by the control signal transmitting / receiving unit 160, the control unit 170 activates the disconnection button (step S118). Specifically, the control unit 170 starts monitoring whether or not the touch on the command area of the cut button specified by the coordinate information included in the cut button activation message is acquired by the user interface unit 180. If the disconnection button is not touched (step S120 / NO), the process returns to step S120 again, and the control unit 170 continues monitoring of the touch on the disconnection button. On the other hand, when the disconnection button is touched (step S120 / YES), the control unit 170 executes the command of the disconnection button indicated by the command information included in the activation message of the disconnection button (step S122). For example, the control unit 170 turns off the microphone 151 to invalidate the voice input. Also, the control unit 170 performs PAUSE setting of Miracast, and temporarily suspends audio data transmission to the source device 200. The control unit 170 also controls the control signal transmission / reception unit 160 to transmit a cooperation request message related to the disconnection button to the source device 200. For example, the control unit 170 transmits, as a cooperation request message, a message including operation information indicating that the disconnection button has been touched, and a PAUSE setting message. As a result, the wireless connection between the source device 200 and the called party is disconnected, and the voice call is ended. Thus, for example, the screen changes from the voice call screen to the menu screen, and the disconnect button is not displayed. Along with this, the source device 200 transmits the disconnection button invalidation message to the sink device 100. When the deactivation signal of the disconnection button is received by the control signal transmission / reception unit 160, the control unit 170 deactivates the disconnection button (step S124). Specifically, the control unit 170 stops monitoring of whether or not the touch on the command area of the cutting button specified by the coordinate information included in the disabling message of the cutting button is acquired by the user interface unit 180.

  After the Miracast connection is established (step S106), when it is determined that the activation message of the calling button is received (step S126 / YES), referring to FIG. 8, the control unit 170 enables the calling button. (Step S132). For example, the control unit 170 starts monitoring whether the touch on the command area of the calling button specified by the coordinate information included in the activation message of the calling button is acquired by the user interface unit 180. .

  If the call button is not touched (NO in step S134), the control unit 170 determines whether the control signal transmission / reception unit 160 receives an invalidation message of the call button (step S148). If it is determined that the call has not been received (step S148 / NO), the process returns to step S134 again, and the control unit 170 continues monitoring for a touch on the call button. On the other hand, when it is determined that the call has been received (step S148 / YES), the control unit 170 invalidates the calling button (step S150). Then, the process returns to step S108 again.

  When the call button is touched (YES in step S134), the control unit 170 executes a command of the call button indicated by the command information included in the activation message of the call button (step S136). For example, the control unit 170 turns on the microphone 151 to enable voice input. Further, the control unit 170 performs PAUSE release setting (PLAY setting) of Miracast, and resumes transmission of audio data to the source device 200. In addition, the control unit 170 controls the control signal transmission / reception unit 160 to transmit a cooperation request message related to the call origination button to the source device 200. For example, the control unit 170 transmits, as a cooperation request message, a message including operation information indicating that the call button has been touched, and a PLAY setting message. Thereby, for example, a wireless connection between the source device 200 and the calling party is established, and a voice call is started. Therefore, for example, the screen changes from the calling screen to the voice call screen, the calling button is not displayed, and the disconnecting button is displayed. Along with this, the source device 200 transmits the invalidation message of the calling button and the activation message of the disconnect button to the sink device 100. When the call signal invalidation message is received by the control signal transmitting / receiving unit 160, the control unit 170 deactivates the call button (step S138).

  In addition, when the activation message of the disconnection button is received by the control signal transmitting / receiving unit 160, the control unit 170 activates the disconnection button (step S140). If the disconnection button is not touched (NO in step S142), the process returns to step S142 again, and the control unit 170 continues monitoring of the touch on the disconnection button. On the other hand, when the disconnection button is touched (step S142 / YES), the control unit 170 executes the disconnection button command indicated by the command information included in the disconnection button activation message (step S144). For example, the control unit 170 turns off the microphone 151 to invalidate the voice input. Also, the control unit 170 performs PAUSE setting of Miracast, and temporarily suspends audio data transmission to the source device 200. Also, the control unit 170 controls the control signal transmission / reception unit 160 to transmit a cooperation request message related to the disconnection button to the source device 200. For example, the control unit 170 transmits, as a cooperation request message, a message including operation information indicating that the disconnection button has been touched, and a PAUSE setting message. As a result, the wireless connection between the source device 200 and the calling party is disconnected, and the voice call is ended. Thus, for example, the screen changes from the voice call screen to the menu screen, and the disconnect button is not displayed. Along with this, the source device 200 transmits the disconnection button invalidation message to the sink device 100. When the deactivation signal of the disconnection button is received by the control signal transmitting / receiving unit 160, the control unit 170 deactivates the disconnection button (step S146).

  The flow of the incoming call processing and the calling processing performed by the sink device 100 according to the present embodiment has been described above. The processes after step S108 may be repeated as necessary. The sink device 100 according to the present embodiment can validate the command area upon receiving the validation message, and can directly execute the command when the command area is manipulated. For example, when the incoming call button, the outgoing call button, and the disconnecting button are touched, the sink device 100 directly performs various processing such as ON / OFF of the microphone 151 without passing through the determination and instruction by the source device 200. be able to. As a result, the number of control messages transmitted to and received from the source device 200 during the time between the touch and the operation of the sink device 100 is reduced, so that the response speed is increased and the power consumption is reduced. .

  Also, by establishing the Miracast connection in both directions in advance, the sink device 100 improves the response speed as compared with the case where the two-way connection is established for the first time when a voice call is required. be able to. In addition, the sink device 100 performs PAUSE setting while voice communication is not being performed, and can perform power saving by performing PAUSE release setting only while voice communication is performed.

4-2-2. Incoming / outgoing operation of source device 200]
FIGS. 9 and 10 are diagrams showing an example of the flow of incoming call processing and outgoing call processing performed by the source device 200 according to the present embodiment.

  Referring to FIG. 9, first, the control unit 270 of the source device 200 performs initialization processing and preparation processing of each device in the source device 200 (step S202). Next, the control unit 270 controls the wireless communication unit 210 to establish a wireless connection with the wireless communication unit 110 of the sink device 100 (step S204). Next, the wireless communication unit 210 bi-directionally establishes a Miracast connection with the sink device 100 (step S206). Specifically, first, the control unit 270 controls the control signal transmission / reception unit 260 to exchange various control messages with the control signal transmission / reception unit 160 of the sink device 100 to perform various negotiations. Then, the control unit 270 controls the stream transmission / reception unit 220 and the compression / decompression unit 230, converts the content acquired by the content acquisition unit 240 into a wireless signal, and transmits the signal to the sink device 100 by the wireless communication unit 210. By this processing, the source device 200 can operate as a source device to reproduce the content transmitted to the sink device. In addition, the control unit 270 controls the stream transmission / reception unit 220 and the compression / decompression unit 230 to decode content data from the wireless signal received from the sink device 100 by the wireless communication unit 210 and output the content data from the content reproduction unit 250. By this processing, the source device 200 can operate as a sink device, and can play back content received from the sink device 100 functioning as the source device. With such a two-way connection, for example, when making a voice call such as a telephone, it becomes possible to reproduce the voice of the other party by the speaker 152 and transmit the user voice acquired by the microphone 151 to the other party. . Here, since the reception of the microphone sound from the sink device 100 is unnecessary while the voice call is not performed, the control unit 270 performs PAUSE setting of Miracast and receives the audio data from the sink device 100. It is preferable to temporarily pause.

  Next, the control unit 270 determines whether or not the incoming call screen is displayed (step S208). The incoming call screen is displayed, for example, from when an external call is made using a public network wireless communication network to when a voice call is started. If it is determined that the incoming call screen is not displayed (NO in step S208), the control unit 270 determines whether the calling screen is displayed (step S226). The calling screen is, for example, a screen for selecting a calling party or a screen for inputting a telephone number when making a call using a public network wireless communication network, and a call can be made by touching the calling button. It is a screen. If it is determined that none of them is displayed (step S226 / NO), the process returns to step S208 again.

  When there is an incoming call from the outside and the incoming call screen is displayed (step S208 / YES), the control unit 270 controls the control signal transmission / reception unit 260 to transmit an activation message of an incoming call button to the sink device 100 (Step S210). Thus, the incoming call button is activated in the sink device 100. Next, the control unit 270 determines whether the incoming call button has been touched on the sink device 100 (step S212). When the cooperation request message resulting from the touch on the incoming call button is received from the sink device 100, the control unit 270 determines that the incoming call button is touched.

  When it is determined that the incoming call button is not touched (step S212 / NO), the control unit 270 determines whether the display of the incoming call screen is continued (step S228). If it is determined that the display of the incoming call screen continues (step S228 / YES), the process returns to step S212 again, and the control unit 270 repeats the determination as to whether or not the incoming call button has been touched. On the other hand, when it is determined that the display of the incoming call screen is finished (step S228 / NO), the control unit 270 controls the control signal transmitting / receiving unit 260 to send an invalidation message of the incoming call button to the sink device 100. It transmits (step S230). As a result, the incoming call button is invalidated in the sink device 100. Then, the process returns to step S208 again.

  If it is determined that the incoming call button has been touched (step S212 / YES), the control unit 270 performs an incoming call process (step S214). For example, based on the cooperation request message, the control unit 270 controls the wireless communication unit 210 to establish a wireless connection with the called party, and starts a voice call between the source device 200 and the called party. In addition, the control unit 270 performs PAUSE release setting based on the message for PLAY setting of Miracast, and resumes reception of audio data from the sink device 100. As a result, the user voice acquired by the microphone 151 of the sink device 100 can be transmitted to the other party. When a voice call is started, for example, the screen changes from the incoming call screen to the voice call screen, the incoming call button is not displayed, and the disconnect button is displayed. Along with this, the control unit 270 controls the control signal transmitting / receiving unit 260 to transmit the invalidation message of the incoming call button and the activation message of the disconnect button to the sink device 100 (steps S216 and S218). In the device 100, the incoming call button is deactivated and the disconnect button is activated.

  Next, the control unit 270 determines whether the disconnection button is touched on the sink device 100 (step S220). When the cooperation request message resulting from the touch on the disconnection button is received from the sink device 100, the control unit 270 determines that the disconnection button is touched. If the disconnection button is not touched (step S220 / NO), the process returns to step S220 again, and the control unit 270 repeats the determination as to whether or not the disconnection button is touched. On the other hand, when it is determined that the disconnection button is touched (step S220 / YES), the control unit 270 performs the disconnection process (step S222). For example, based on the cooperation request message, the control unit 270 controls the wireless communication unit 210 to disconnect the wireless connection with the called party. Further, the control unit 270 performs PAUSE setting based on the message for setting the pause of Miracast, and suspends the reception of audio data from the sink device 100. When the voice call ends in this way, for example, the screen changes from the voice call screen to the menu screen, and the disconnect button is not displayed. Along with this, the control unit 270 controls the control signal transmission / reception unit 260 to transmit a disconnection button invalidation message to the sink device 100 (step S224). As a result, the disconnect button is invalidated in the sink device 100.

  After the Miracast connection is established (step S206), when the calling screen is displayed (step S226 / YES), referring to FIG. 10, the control unit 270 controls the control signal transmitting / receiving unit 260 to call the call button. The activation message of is transmitted to the sink device 100 (step S232). As a result, in the sink device 100, the call origination button is activated. Next, the control unit 270 determines whether the call button is touched on the sink device 100 (step S234). When the cooperation request message resulting from the touch on the call origination button is received from the sink device 100, the control unit 270 determines that the call origination button is touched.

  If it is determined that the call button is not touched (NO in step S234), the control unit 270 determines whether the display of the call screen is continued (step S248). If it is determined that the display of the calling screen is continuing (step S248 / YES), the process returns to step S234 again, and the control unit 270 repeats the determination as to whether or not the calling button has been touched. On the other hand, when it is determined that the display of the calling screen is finished, for example, when the user stops calling or the like (step S248 / NO), control unit 270 controls control signal transmitting / receiving unit 260, The message for disabling the call button is sent to the sink device 100 (step S250). As a result, in the sink device 100, the call origination button is invalidated. Then, the process returns to step S208 again.

  If it is determined that the call button has been touched (YES in step S234), control unit 270 performs a call process (step S236). For example, based on the cooperation request message, the control unit 270 controls the wireless communication unit 210 to establish a wireless connection with the calling party, and starts a voice call between the source device 200 and the calling party. In addition, the control unit 270 performs PAUSE release setting based on the message for PLAY setting of Miracast, and resumes reception of audio data from the sink device 100. As a result, the user voice acquired by the microphone 151 of the sink device 100 can be transmitted to the other party. When a voice call is started, for example, the screen changes from the calling screen to the voice call screen, the calling button is not displayed, and the disconnection button is displayed. Along with this, the control unit 270 controls the control signal transmitting / receiving unit 260 to transmit an invalidation message of the calling button and an activation message of the disconnect button to the sink device 100 (steps S238 and S240). In the device 100, the call out button is deactivated and the disconnect button is activated.

  Next, the control unit 270 determines whether the disconnection button is touched on the sink device 100 (step S 242). When the cooperation request message resulting from the touch on the disconnection button is received from the sink device 100, the control unit 270 determines that the disconnection button is touched. If the disconnection button is not touched (NO in step S242), the process returns to step S220 again, and the control unit 270 repeats the determination as to whether or not the disconnection button is touched. On the other hand, when it is determined that the disconnection button is touched (step S242 / YES), the control unit 270 performs the disconnection process (step S244). For example, based on the cooperation request message, the control unit 270 controls the wireless communication unit 210 to disconnect the wireless connection with the calling party. Further, the control unit 270 performs PAUSE setting based on the message for setting the pause of Miracast, and suspends the reception of audio data from the sink device 100. When the voice call ends in this way, for example, the screen changes from the voice call screen to the menu screen, and the disconnect button is not displayed. Along with this, the control unit 270 controls the control signal transmission / reception unit 260 to transmit a disconnection button invalidation message to the sink device 100 (step S246). As a result, the disconnect button is invalidated in the sink device 100.

  The flow of the call reception process and the call generation process performed by the source device 200 according to the present embodiment has been described above. The processes after step S208 may be repeated as necessary. The source device 200 according to the present embodiment enables direct command execution in the sink device 100 by transmitting an enabling message. Therefore, since the number of control messages transmitted to and received from the sink device 100 is reduced after the user operation is acquired in the sink device 100 until the operation, the response speed is increased, and the power consumption is increased. Is reduced.

[4-3. Flow of message sending and receiving]
Subsequently, the correspondence between the control messages transmitted and received by the sink device 100 and the source device 200 and the operations of the respective devices described above with reference to FIGS. 7 to 10 will be described with reference to FIGS. .

(Calling process)
FIG. 11 is a sequence diagram showing an example of the flow of call processing performed in the communication system 1 according to the present embodiment.

  As shown in FIG. 11, first, the sink device 100 and the source device 200 bidirectionally establish Miracast connection using WFD (Wi-Fi Direct) (step S302). At this time, the sink device 100 performs PAUSE setting of Miracast regarding transmission of audio data acquired by the microphone 151 of the sink device 100, and suspends transmission of audio data from the sink device 100 to the source device 200 (step S304). ).

  When the calling screen is displayed on the sink device 100 and the source device 200 (step S306), the source device 200 transmits a call button activation message to the sink device 100 (step S308). As a result, the call origination button is activated in the sink device 100. Here, with reference to FIG. 12, an example of the calling screen and the calling button will be described.

FIG. 12 is a diagram showing an example of a call origination screen and a call origination button. The left side of FIG. 12 is a screen for inputting a telephone number, and is a calling screen capable of making a call to the input telephone number when the call button 300a is touched. The right side of FIG. 12 is a screen for displaying the outgoing call history, and is a outgoing call screen which can be sent to the telephone number indicated by the outgoing call history when the outgoing call history 300 b or the outgoing call button 300 c is touched. In Figure 12, call button 300a, 300 c, and call history 300b functions as a call button. As shown in the right figure of FIG. 12, the command area may be plural.

  When the touch on the call button is recognized in the sink device 100 (step S310), the sink device 100 transmits a cooperation request message to the source device 200. For example, the sink device 100 transmits, using the UIBC, operation information indicating that the call button has been touched (step S312). In addition, the sink device 100 performs PAUSE release setting for transmission of audio data acquired by the microphone 151 of the sink device 100, and resumes audio data transmission to the source device 200 (step S314). Next, in response to the recognition of the touch on the call button, the sink device 100 turns on the microphone 151 to validate the voice input (step S316a). On the other hand, the source device 200 performs call processing based on the received cooperation request message (step S316b). Then, the source device 200 sinks the callout button deactivation message and the disconnection button activation message in accordance with the display of the callout button hiding and disconnection button by the transition from the callout screen to the voice call screen. It transmits to the apparatus 100 (steps S318 and S320). As a result, a voice call is made with the sink device 100 and the source device 200 with the calling party (step S322). Here, an example of the voice call screen and the disconnect button will be described with reference to FIG.

  FIG. 13 is a diagram showing an example of a voice call screen and a disconnect button. On the voice call screen shown in FIG. 13, the other party in the voice call is displayed, and the disconnect button 310 is also displayed. When the disconnect button 310 is touched, the voice call ends.

  When the touch on the disconnection button is recognized in the sink device 100 (step S324), the sink device 100 transmits a cooperation request message to the source device 200. For example, the sink device 100 transmits operation information indicating that the disconnection button has been touched, using the UIBC (step S326). Also, the sink device 100 performs PAUSE setting for transmission of audio data acquired by the microphone 151 of the sink device 100, and suspends audio data transmission to the source device 200 (step S328). Next, in response to the recognition of the touch on the disconnection button, the sink device 100 turns off the microphone 151 to invalidate the voice input (step S330a). On the other hand, the source device 200 performs the disconnection process based on the received cooperation request message (step S330b). Then, the source device 200 transmits a disconnection button invalidation message to the sink device 100 in accordance with the termination of the voice call screen and the non-display of the disconnection button (step S332). As a result, the voice communication between the sink device 100 and the source device 200 and the calling party ends.

  Although the example in which the disconnection of the voice call is performed by the sink device 100 is described in FIG. The process in such a case will be described with reference to FIG.

  FIG. 14 is a sequence diagram showing an example of the flow of call processing performed in the communication system 1 according to the present embodiment. The processes in steps S302 to S322 in FIG. 14 are the same as those described above with reference to FIG.

  As shown in FIG. 14, when the other party disconnects the voice call (step S334), the source device 200 performs disconnection processing (step S336). Next, the source device 200 performs PAUSE setting for the reception of the audio data acquired by the microphone 151 of the sink device 100, and temporarily suspends the audio data reception from the sink device 100 (step S338). Next, upon receiving PAUSE, the sink device 100 turns off the microphone 151 to invalidate the voice input (step S340). Then, the source device 200 transmits an invalidation message of the disconnect button to the sink device 100 in accordance with the termination of the voice call screen and the non-display of the disconnect button (step S 342). As a result, the voice communication between the sink device 100 and the source device 200 and the calling party ends.

  The correspondence between control messages transmitted and received regarding the calling process and the operation of each device has been described above. Subsequently, with reference to FIG. 15 to FIG. 17, the correspondence between control messages transmitted and received in connection with an incoming call process and operations of respective devices will be described.

(Incoming call processing)
FIG. 15 is a sequence diagram showing an example of the flow of call reception processing executed in the communication system 1 according to the present embodiment.

  As shown in FIG. 15, first, the sink device 100 and the source device 200 bi-directionally establish Miracast connection using the WFD (step S402). At this time, the sink device 100 performs PAUSE setting of Miracast regarding transmission of audio data acquired by the microphone 151 of the sink device 100, and suspends transmission of audio data from the sink device 100 to the source device 200 (step S404). ).

  When the incoming call screen is displayed on the sink device 100 and the source device 200 (step S406), the source device 200 transmits an incoming call button activation message to the sink device 100 (step S408). Thus, the incoming call button is activated in the sink device 100. Here, an example of the incoming call screen and the incoming call button will be described with reference to FIG.

  FIG. 16 is a diagram showing an example of an incoming call screen and an incoming call button. As shown in FIG. 16, the called party as the originator of the telephone is displayed, and the incoming call button 400 is displayed. In this example, when the icon on the left side of the incoming call button 400 is touched and swiped to the right, voice communication with the called party becomes possible. Of course, the incoming call button may be capable of making a voice call with the called party if it is simply touched.

  When the touch (swipe) on the incoming call button is recognized in the sink device 100 (step S410), the sink device 100 transmits a cooperation request message to the source device 200. For example, the sink device 100 transmits operation information indicating that the incoming call button has been touched, using the UIBC (step S412). In addition, the sink device 100 performs PAUSE release setting for transmission of audio data acquired by the microphone 151 of the sink device 100, and resumes audio data transmission to the source device 200 (step S414). Next, in response to the recognition of the touch on the incoming call button, the sink device 100 turns on the microphone 151 to validate the voice input (step S416a). Meanwhile, the source device 200 performs an incoming call process based on the received cooperation request message (step S416b). Then, the source device 200 sinks the invalidation message of the incoming call button and the activation message of the disconnect button in accordance with the display of the incoming call button and the display of the disconnect button by the transition from the incoming call screen to the voice communication screen. It transmits to the apparatus 100 (steps S418 and S420). As a result, a voice call is made between the sink device 100 and the source device 200 and the called party (step S422). Specific examples of the voice call screen and the disconnect button are as described above, for example, with reference to FIG.

  When the touch on the disconnection button is recognized in the sink device 100 (step S424), the sink device 100 transmits a cooperation request message to the source device 200. For example, the sink device 100 transmits operation information indicating that the disconnection button has been touched, using the UIBC (step S426). In addition, the sink device 100 performs PAUSE setting for transmission of audio data acquired by the microphone 151 of the sink device 100, and suspends audio data transmission to the source device 200 (step S428). Next, in response to the recognition of the touch on the disconnection button, the sink device 100 turns off the microphone 151 to invalidate the voice input (step S430a). On the other hand, the source device 200 performs the disconnection process based on the received cooperation request message (step S430b). Then, the source device 200 transmits the disconnection button invalidation message to the sink device 100 in accordance with the termination of the voice call screen and the non-display of the disconnection button (step S432). As a result, the voice communication between the sink device 100 and the source device 200 and the called party ends.

  Although the example in which the disconnection of the voice call is performed by the sink device 100 has been described with reference to FIG. The process in such a case will be described with reference to FIG.

  FIG. 17 is a sequence diagram showing an example of the flow of call reception processing executed in the communication system 1 according to the present embodiment. The processes in steps S402 to S422 in FIG. 17 are as described above with reference to FIG.

  As shown in FIG. 17, when the other party disconnects the voice call (step S434), the source device 200 performs disconnection processing (step S436). Next, the source device 200 performs PAUSE setting for the reception of the audio data acquired by the microphone 151 of the sink device 100, and temporarily suspends the audio data reception from the sink device 100 (step S438). Next, upon receiving PAUSE, the sink device 100 turns off the microphone 151 to invalidate the voice input (step S440). Then, the source device 200 transmits the disconnection button invalidation message to the sink device 100 in accordance with the termination of the voice call screen and the non-display of the disconnection button (step S 442). As a result, the voice communication between the sink device 100 and the source device 200 and the called party ends.

  The control messages transmitted and received in the incoming call process and the outgoing call process and the correspondence between the operations of the respective devices have been described above.

[4-4. Supplement]
18 to 23 show the sequence of messages defined in Miracast. In FIG. 18, after the sink device (Sink) and the source device (Source) start connection with WFD, transmission of content by the source device and reproduction of content in the sink device are performed, and then disconnection is performed. It provides an overview of the message sequence. FIG. 19 shows the detailed RTSP message sequence of the capability negotiation procedure (WFD Capability Negotiation) in FIG. Also, FIG. 20 shows a detailed RTSP message sequence of the session establishment procedure (WFD Session Establishment) in FIG. FIG. 21 shows an RTSP message sequence for performing PAUSE setting for content transmission. FIG. 22 shows an RTSP message sequence for performing PAUSE release setting (PLAY setting) for content transmission / reception. FIG. 23 shows the RTBC message sequence of the UIBC. In addition, since these message sequences are as the specification of the Wi-Fi display shown by the said nonpatent literature 1, the detailed description regarding each message is abbreviate | omitted.

<5. Second embodiment>
In the present embodiment, the sink device 100 performs various processes under the control of the source device 200 without using the operation setting message. The operation processing according to the present embodiment will be described below with reference to FIGS. 24 to 27.

(Calling process)
FIG. 24 is a sequence diagram showing an example of the flow of call processing performed in the communication system 1 according to the present embodiment.

  As shown in FIG. 24, first, the sink device 100 and the source device 200 bi-directionally establish a Miracast connection using the WFD (step S502). At this time, the sink device 100 performs PAUSE setting of Miracast regarding transmission of audio data acquired by the microphone 151 of the sink device 100, and suspends transmission of audio data from the sink device 100 to the source device 200 (step S503). ). The PAUSE setting may be performed from the source device 200. When the calling screen is displayed (step S504) and the touch on the calling button is recognized in the sink device 100 (step S506), the sink device 100 displays operation information indicating that the calling button has been touched, It transmits to the source device 200 using UIBC as a cooperation request message (step S508). Next, the source device 200 recognizes that the call button has been touched in the sink device 100 based on the cooperation request message, and performs call processing (step S510). Next, the source device 200 transmits, to the sink device 100, a control message instructing execution of the microphone ON process (step S512). In addition, the source device 200 performs PAUSE release setting for the reception of the audio data acquired by the microphone 151 of the sink device 100, and resumes the audio data reception from the sink device 100 (step S513). Next, in accordance with the received control message, the sink device 100 turns on the microphone 151 to validate voice input (step S514). Thus, a voice call is made between the sink device 100 and the source device 200 and the calling party (step S516).

  When the touch on the disconnect button is recognized in the sink device 100 (step S518), the sink device 100 transmits the operation information indicating that the disconnect button is touched to the source device 200 using the UIBC as a cooperation request message. (Step S520). Next, the source device 200 recognizes that the disconnection button has been touched in the sink device 100 based on the cooperation request message, and performs disconnection processing (step S522). Next, the source device 200 transmits, to the sink device 100, a control message instructing execution of the microphone OFF process (step S524). In addition, the source device 200 performs PAUSE setting for the reception of audio data acquired by the microphone 151 of the sink device 100, and temporarily suspends the audio data reception from the sink device 100 (step S525). Next, in accordance with the received control message, the sink device 100 turns off the microphone 151 to invalidate the voice input (step S526). Thus, the voice call between the sink device 100 and the source device 200 and the calling party is ended. Note that the control message of the execution instruction of the microphone ON / OFF processing and PAUSE setting / cancel are merged, and when PAUSE is set, the sink device 100 performs microphone OFF processing, and when PAUSE is released, the sink device 100 performs microphone ON processing. You may do so.

  Although the example in which the disconnection of the voice call is performed by the sink device 100 has been described with reference to FIG. 24, it may be considered that the other party may perform the disconnection of the voice call. The process in such a case will be described with reference to FIG.

  FIG. 25 is a sequence diagram showing an example of the flow of call processing performed in the communication system 1 according to the present embodiment. The processing in steps S502 to S516 in FIG. 25 is as described above with reference to FIG. 24, and thus the description thereof is omitted here.

  As shown in FIG. 25, when the other party disconnects the voice call (step S528), the source device 200 performs disconnection processing (step S530). Next, the source device 200 transmits, to the sink device 100, a control message instructing to execute the microphone OFF process (step S532). In addition, the source device 200 performs PAUSE setting for the reception of audio data acquired by the microphone 151 of the sink device 100, and suspends the reception of audio data from the sink device 100 (step S533). Next, in accordance with the received control message, the sink device 100 turns off the microphone 151 to invalidate the voice input (step S534). Thus, the voice call between the sink device 100 and the source device 200 and the calling party is ended. Note that the control message of the execution instruction of the microphone ON / OFF processing and PAUSE setting / cancel are merged, and when PAUSE is set, the sink device 100 performs microphone OFF processing, and when PAUSE is released, the sink device 100 performs microphone ON processing. You may do so.

  Heretofore, an example of the flow of call processing according to the present embodiment has been described. Subsequently, an example of the flow of call reception processing according to the present embodiment will be described with reference to FIGS.

(Incoming call processing)
FIG. 26 is a sequence diagram showing an example of the flow of call reception processing executed in the communication system 1 according to the present embodiment.

  As shown in FIG. 26, first, the sink device 100 and the source device 200 establish a Miracast connection in both directions by using the WFD (step S602). At this time, the sink device 100 performs PAUSE setting of Miracast regarding transmission of audio data acquired by the microphone 151 of the sink device 100, and temporarily suspends transmission of audio data from the sink device 100 to the source device 200 (step S603). ). The PAUSE setting may be performed from the source device 200. When the incoming call screen is displayed (step S604) and the touch on the incoming call button is recognized in the sink device 100 (step S606), the sink device 100 displays the operation information indicating that the incoming call button has been touched, It transmits to the source device 200 using UIBC as a cooperation request message (step S608). Next, the source device 200 recognizes that the incoming call button has been touched in the sink device 100 based on the cooperation request message, and performs an incoming call process (step S610). Next, the source device 200 transmits a control message instructing to execute the microphone ON process to the sink device 100 (step S612). In addition, the source device 200 performs PAUSE release setting for the reception of the audio data acquired by the microphone 151 of the sink device 100, and resumes the audio data reception from the sink device 100 (step S613). Next, in accordance with the received control message, the sink device 100 turns on the microphone 151 to validate the voice input (step S614). Thus, a voice call is made between the sink device 100 and the source device 200 and the called party (step S616).

  When the touch on the disconnection button is recognized in the sink device 100 (step S618), the sink device 100 transmits the operation information indicating that the disconnection button is touched to the source device 200 using the UIBC as a cooperation request message. (Step S620). Next, the source device 200 recognizes that the disconnection button has been touched in the sink device 100 based on the cooperation request message, and performs disconnection processing (step S 622). Next, the source device 200 transmits, to the sink device 100, a control message instructing execution of the microphone OFF process (step S624). Also, the source device 200 performs PAUSE setting for the reception of the audio data acquired by the microphone 151 of the sink device 100, and suspends the reception of the audio data from the sink device 100 (step S625). Next, in accordance with the received control message, the sink device 100 turns off the microphone 151 to invalidate the voice input (step S 626). Thus, the voice call between the sink device 100 and the source device 200 and the called party is ended. Note that the control message of the execution instruction of the microphone ON / OFF processing and PAUSE setting / cancel are merged, and when PAUSE is set, the sink device 100 performs microphone OFF processing, and when PAUSE is released, the sink device 100 performs microphone ON processing. You may do so.

  Although the example in which the disconnection of the voice call is performed by the sink device 100 has been described in FIG. The process in such a case will be described with reference to FIG.

  FIG. 27 is a sequence diagram showing an example of the flow of call reception processing executed in the communication system 1 according to the present embodiment. The processes in steps S602 to S616 in FIG. 27 are as described above with reference to FIG. 26, and thus the description thereof is omitted here.

  As shown in FIG. 27, when the other party disconnects the voice call (step S628), the source device 200 performs disconnection processing (step S630). Next, the source device 200 transmits, to the sink device 100, a control message instructing to execute the microphone OFF process (step S632). In addition, the source device 200 performs PAUSE setting for the reception of audio data acquired by the microphone 151 of the sink device 100, and suspends the reception of audio data from the sink device 100 (step S633). Next, in accordance with the received control message, the sink device 100 turns off the microphone 151 to invalidate the voice input (step S634). Thus, the voice call between the sink device 100 and the source device 200 and the called party is ended. Note that the control message of the execution instruction of the microphone ON / OFF processing and PAUSE setting / cancel are merged, and when PAUSE is set, the sink device 100 performs microphone OFF processing, and when PAUSE is released, the sink device 100 performs microphone ON processing. You may do so.

  The operation processing according to the present embodiment has been described above. According to the present embodiment, the sink device 100 can be turned off while the microphone 151 is not used under the control of the source device 200, and compared to the case where the sink device 100 is always turned on, the power consumption is reduced. Is reduced. When the incoming call button, the outgoing call button or the disconnect button is directly operated on the source device 200 side, the sink device 100 can operate under the control of the source device 200 as in the above-described message sequence. .

<6. Third embodiment>
In the first and second embodiments, the sink device 100 and the source device 200 operate in cooperation with each other. In the present embodiment, the sink device 100 performs its own operation without linking with the source device 200.

  Specifically, the source device 200 according to the present embodiment transmits, as command information, an operation setting message including information indicating a process of changing the environment setting of the sink device 100 to the sink device 100. The environment setting includes various setting items such as volume, brightness of screen, time setting, power saving setting, and the like. When the process indicated by the command information is a process of changing the environment setting, the sink device 100 cancels the transmission of the cooperation request message in the case where the operation to the command area is recognized. That is, the sink device 100 does not transmit the cooperation request message even if the command is executed. Therefore, in the communication system 1 according to the present embodiment, when the operation to change the environment setting is performed on the sink device 100 side, the setting change is reflected on the sink device 100 side, but the operation on the source device 200 side is performed. Configuration change is not reflected because it can not be recognized.

  Similarly, when the source device 200 recognizes an operation to change the environment setting, the source device 200 does not transmit a control message for causing the sink device 100 to execute the process to change the environment setting to the sink device 100. Therefore, in the communication system 1 according to the present embodiment, when the operation to change the environment setting is performed on the source device 200 side, the setting change is reflected on the source device 200 side, but the operation on the sink device 100 side is performed. Configuration change is not reflected because it can not be recognized.

  For example, when the user operates the in-vehicle monitor 100 to adjust the volume, the user intends to adjust the volume on the in-vehicle monitor 100 side and does not intend to adjust even the volume on the smartphone 200 side. . In the present embodiment, among the apparatuses included in the communication system 1, only the apparatus for which the user has performed an operation to change the environment setting reflects the setting change, so that the setting change according to the user's intention can be realized. .

  Here, as an example, a display example of the environment setting screen according to the present embodiment and a specific example of the operation setting message will be described with reference to FIGS. 28 and 29. FIG. 28 and FIG. 29 are views showing display examples of the environment setting screen according to the present embodiment. In the environment setting screen shown in FIG. 28, when the volume setting area 700, 710, 720 is swiped, the sink device 100 or the source device 200 only controls the volume of the operated device according to the swipe direction and amount. It is possible to adjust the An example of the operation setting message regarding the environment setting screen shown in FIG. 28 is shown below.

<Activation message>
Coordinate information: (X1, Y1), (X2, Y2)
Type information: (swipe)
Command information: (volume)
Flag information: (valid)

<Invalidation message>
Coordinate information: (X1, Y1), (X2, Y2)
Type information: (swipe)
Command information: (volume)
Flag information: (disabled)

  In the environment setting screen shown in FIG. 29, when the brightness setting area 730 is swiped, the sink device 100 or the source device 200 displays the screen of only the operated device according to the direction and amount of the swipe. It is possible to adjust the brightness. An example of the operation setting message regarding the environment setting screen shown in FIG. 29 is shown below.

<Activation message>
Coordinate information: (X1, Y1), (X2, Y2)
Type information: (swipe)
Command information: (brightness)
Flag information: (valid)

<Invalidation message>
Coordinate information: (X1, Y1), (X2, Y2)
Type information: (swipe)
Command information: (brightness)
Flag information: (disabled)

  In the present embodiment, among the apparatuses included in the communication system 1, only the apparatus for which the user has performed an operation to change the environment setting reflects the setting change, so that the setting change according to the user's intention can be realized. . In the above description, only the device of the sink device 100 or the source device 200 that has accepted the operation has been described as the operation for changing the environment setting, but the present embodiment is not limited to this. For example, in accordance with the user setting, even in the operation of changing the environment setting, the setting change may be reflected on both the sink device 100 and the source device 200.

<7. Application example>
The technology according to the present disclosure can be applied to various products. For example, the sink device 100 and the source device 200 may be fixed devices such as a smartphone, a tablet PC (Personal Computer), a notebook PC, a mobile terminal such as a portable game terminal or a digital camera, a television receiver, a printer, a digital scanner or a network storage The present invention may be realized as a terminal or an on-vehicle terminal such as a car navigation device. In addition, the sink device 100 and the source device 200 are terminals (MTC (Machine Type Communication) terminals such as smart meters, vending machines, remote monitoring devices or POS (Point Of Sale) terminals that perform M2M (Machine To Machine) communication. May be realized. Furthermore, the sink device 100 and the source device 200 may be wireless communication modules (for example, integrated circuit modules configured by one die) mounted on these terminals.

7-1. First application example]
FIG. 30 is a block diagram showing an example of a schematic configuration of a smartphone 900 to which the technology according to the present disclosure can be applied. The smartphone 900 includes a processor 901, a memory 902, a storage 903, an external connection interface 904, a camera 906, a sensor 907, a microphone 908, an input device 909, a display device 910, a speaker 911, a wireless communication interface 913, an antenna switch 914, an antenna 915, A bus 917, a battery 918 and an auxiliary controller 919 are provided.

  The processor 901 may be, for example, a central processing unit (CPU) or a system on chip (SoC), and controls functions of an application layer and other layers of the smartphone 900. The memory 902 includes a random access memory (RAM) and a read only memory (ROM), and stores programs and data to be executed by the processor 901. The storage 903 may include a storage medium such as a semiconductor memory or a hard disk. The external connection interface 904 is an interface for connecting an external device such as a memory card or a USB (Universal Serial Bus) device to the smartphone 900.

  The camera 906 includes an imaging element such as, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and generates a captured image. The sensor 907 may include, for example, a sensor group such as a positioning sensor, a gyro sensor, a geomagnetic sensor, and an acceleration sensor. The microphone 908 converts audio input to the smartphone 900 into an audio signal. The input device 909 includes, for example, a touch sensor that detects a touch on the screen of the display device 910, a keypad, a keyboard, a button, a switch, or the like, and receives an operation or information input from the user. The display device 910 has a screen such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display, and displays an output image of the smartphone 900. The speaker 911 converts an audio signal output from the smartphone 900 into an audio.

  The wireless communication interface 913 supports one or more of wireless LAN standards such as IEEE 802.11a, 11 b, 11 g, 11 n, 11 ac, and 11 ad to execute wireless communication. The wireless communication interface 913 may communicate with other devices via the wireless LAN access point in the infrastructure mode. In addition, the wireless communication interface 913 can directly communicate with another device in a direct communication mode such as an ad hoc mode or Wi-Fi Direct. In Wi-Fi Direct, one of two terminals operates as an access point unlike the ad hoc mode, but communication is directly performed between the terminals. The wireless communication interface 913 may typically include a baseband processor, an RF (Radio Frequency) circuit, a power amplifier, and the like. The wireless communication interface 913 may be a memory that stores a communication control program, a processor that executes the program, and a one-chip module in which related circuits are integrated. The wireless communication interface 913 may support other types of wireless communication schemes such as a near field wireless communication scheme, a close proximity wireless communication scheme, or a cellular communication scheme in addition to the wireless LAN scheme. The antenna switch 914 switches the connection destination of the antenna 915 between a plurality of circuits (for example, circuits for different wireless communication schemes) included in the wireless communication interface 913. The antenna 915 has a single or a plurality of antenna elements (for example, a plurality of antenna elements constituting a MIMO antenna), and is used for transmission and reception of a wireless signal by the wireless communication interface 913.

  In addition, it is not limited to the example of FIG. 30, The smart phone 900 may be equipped with a some antenna (For example, the antenna for wireless LANs, the antenna for proximity wireless communication systems, etc.). In that case, the antenna switch 914 may be omitted from the configuration of the smartphone 900.

  The bus 917 connects the processor 901, the memory 902, the storage 903, the external connection interface 904, the camera 906, the sensor 907, the microphone 908, the input device 909, the display device 910, the speaker 911, the wireless communication interface 913 and the auxiliary controller 919 to one another. . The battery 918 supplies power to each block of the smartphone 900 shown in FIG. 30 via a feed line partially shown by a broken line in the figure. The auxiliary controller 919 operates minimum necessary functions of the smartphone 900, for example, in the sleep mode.

  The smartphone 900 illustrated in FIG. 30 can operate as a sink device. In that case, the smartphone 900 may have the function of the sink device 100 described using, for example, FIG. 2. Thus, the smartphone 900 operates directly based on the operation setting message, so that the processing according to the user operation can be directly executed without the control by the source device. In addition, the smartphone 900 can operate as a source device. In that case, the smartphone 900 may have the function of the source device 200 described using, for example, FIG. 4. Thereby, the smartphone 900 transmits an operation setting message to enable the unique operation of the sink device.

7-2. Second application example]
FIG. 31 is a block diagram showing an example of a schematic configuration of a car navigation device 920 to which the technology according to the present disclosure can be applied. The car navigation device 920 includes a processor 921, a memory 922, a GPS (Global Positioning System) module 924, a sensor 925, a data interface 926, a content player 927, a storage medium interface 928, an input device 929, a display device 930, a speaker 931, wireless communication. An interface 933, an antenna switch 934, an antenna 935 and a battery 938 are provided.

  The processor 921 may be, for example, a CPU or an SoC, and controls the navigation function and other functions of the car navigation device 920. The memory 922 includes a RAM and a ROM, and stores programs and data to be executed by the processor 921.

  The GPS module 924 uses GPS signals received from GPS satellites to measure the location (eg, latitude, longitude and altitude) of the car navigation device 920. The sensor 925 may include, for example, a sensor group such as a gyro sensor, a geomagnetic sensor, and an air pressure sensor. The data interface 926 is connected to the on-vehicle network 941 via, for example, a terminal (not shown), and acquires data generated on the vehicle side, such as vehicle speed data.

  Content player 927 plays content stored on a storage medium (eg, CD or DVD) inserted into storage medium interface 928. The input device 929 includes, for example, a touch sensor, a button, or a switch that detects a touch on the screen of the display device 930, and receives an operation or an information input from a user. The display device 930 has a screen such as an LCD or an OLED display, and displays an image of the navigation function or the content to be reproduced. The speaker 931 outputs the sound of the navigation function or the content to be reproduced.

  The wireless communication interface 933 supports one or more of wireless LAN standards such as IEEE 802.11a, 11 b, 11 g, 11 n, 11 ac, and 11 ad, and performs wireless communication. The wireless communication interface 933 may communicate with other devices via the wireless LAN access point in the infrastructure mode. Also, the wireless communication interface 933 can directly communicate with another device in a direct communication mode such as an ad hoc mode or Wi-Fi Direct. The wireless communication interface 933 may typically include a baseband processor, an RF circuit, a power amplifier, and the like. The wireless communication interface 933 may be a one-chip module in which a memory for storing a communication control program, a processor for executing the program, and related circuits are integrated. The wireless communication interface 933 may support other types of wireless communication schemes, such as a short distance wireless communication scheme, a close proximity wireless communication scheme, or a cellular communication scheme, in addition to the wireless LAN scheme. The antenna switch 934 switches the connection destination of the antenna 935 among a plurality of circuits included in the wireless communication interface 933. The antenna 935 has a single or multiple antenna elements, and is used for transmission and reception of wireless signals by the wireless communication interface 933.

  In addition, it is not limited to the example of FIG. 31, Car navigation apparatus 920 may be provided with a plurality of antennas. In that case, the antenna switch 934 may be omitted from the configuration of the car navigation device 920.

  The battery 938 supplies power to each block of the car navigation device 920 shown in FIG. 31 via a feed line partially shown by a broken line in the figure. The battery 938 also stores power supplied from the vehicle side.

  The car navigation device 920 shown in FIG. 31 can operate as a sink device. In that case, the car navigation device 920 may have the function of the sink device 100 described using, for example, FIG. Thus, the car navigation device 920 can directly execute the processing according to the user's operation without being controlled by the source device by operating based on the operation setting message. Also, the car navigation device 920 can operate as a source device. In that case, the car navigation device 920 may have the function of the source device 200 described using, for example, FIG. 4. Thus, the car navigation device 920 enables unique operation by the sink device by transmitting an operation setting message.

  Further, the technology according to the present disclosure may be realized as an on-board system (or vehicle) 940 including one or more blocks of the car navigation device 920 described above, an on-board network 941, and a vehicle-side module 942. The vehicle-side module 942 generates vehicle-side data such as a vehicle speed, an engine speed, or failure information, and outputs the generated data to the in-vehicle network 941.

<8. Summary>
So far, embodiments of the technology according to the present disclosure have been described in detail using FIGS. According to the above-described embodiment, the sink device displays the image received from the source device, and the operation setting message indicates when the operation to the command area indicated by the operation setting message is acquired among the displayed images. Execute a command. As a result, the sink device can directly execute processing according to the user operation without requiring recognition of a user operation by the source device and an instruction to the sink device. Therefore, since the number of messages exchanged between the sink device and the source device after the user operation is recognized in the sink device is reduced, the response speed is increased and the power consumption is reduced.

  The operation setting message can include at least one or more pieces of operation setting information including a combination of coordinate information indicating a command area, type information indicating the type of operation to be acquired, and command information. Accordingly, the sink device can recognize various operations in one screen and directly execute various commands. Further, since the operation setting message can designate valid or invalid of the operation setting information, the sink device can switch the user operation to be recognized according to the screen transition.

  In addition, by establishing the Miracast connection in both directions in advance, the sink device and the source device can respond faster than when establishing a two-way connection when it is necessary to send and receive content. Can be In addition, the sink device and the source device can realize low power consumption by performing PAUSE setting while transmission / reception of content is not performed and setting PAUSE release only while transmission / reception of content is performed. .

  The preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It will be apparent to those skilled in the art of the present disclosure that various modifications and alterations can be conceived within the scope of the technical idea described in the claims. It is naturally understood that the technical scope of the present disclosure is also included.

  Note that the series of processes by each device described in the present specification may be realized using any of software, hardware, and a combination of software and hardware. The programs that configure the software are stored in advance in, for example, storage media (non-transitory media) provided inside or outside each device. Then, for example, each program is read into the RAM at the time of execution and executed by a processor such as a CPU.

  In addition, the processes described using the flowchart and the sequence diagram in the present specification may not necessarily be performed in the illustrated order. Several processing steps may be performed in parallel. Also, additional processing steps may be employed and some processing steps may be omitted.

  In addition, the effects described in the present specification are merely illustrative or exemplary, and not limiting. That is, the technology according to the present disclosure can exhibit other effects apparent to those skilled in the art from the description of the present specification, in addition to or instead of the effects described above.

The following configurations are also within the technical scope of the present disclosure.
(1)
A wireless communication unit that performs wireless communication with another information processing apparatus;
A display unit for displaying an image received from the other information processing apparatus by the wireless communication unit;
An acquisition unit configured to acquire an operation on a region indicated by control information received from the other information processing apparatus by the wireless communication unit among the images displayed by the display unit;
A control unit configured to perform processing indicated by the control information according to the operation acquired by the acquisition unit;
An information processing apparatus comprising:
(2)
The control information is information for specifying an area where the acquisition unit should acquire an operation, information indicating a type of operation that the acquisition unit should acquire, and the control unit controls when the operation to the area is received. The information processing apparatus according to (1), including a combination of information indicating the process to be performed.
(3)
The information processing apparatus according to (2), wherein the control information includes information indicating whether the combination is valid or invalid.
(4)
The information processing apparatus according to (2) or (3), wherein the control information includes a plurality of the combinations.
(5)
The information according to any one of (1) to (4), wherein the wireless communication unit bidirectionally establishes a connection by Wi-Fi CERTIFIED MIRACAST (registered trademark) with the information processing apparatus. Processing unit.
(6)
When the control unit causes the other information processing apparatus to execute a process corresponding to the process indicated by the control information, the control unit causes the other information processing apparatus to execute a message for executing the process via the wireless communication unit. The information processing apparatus according to any one of (1) to (5), which transmits to the information processing apparatus.
(7)
The message includes operation information indicating the operation acquired by the acquisition unit, and is transmitted using a User Input Back Channel (UIBC) on TCP / IP (Transmission Control Protocol / Internet Protocol), The information processing apparatus according to the above.
(8)
The information processing apparatus according to (6) or (7), wherein the message includes a message for PAUSE setting or PLAY setting of Real Time Streaming Protocol (RTSP).
(9)
The information processing apparatus includes an audio input / output unit that inputs and outputs audio.
The process indicated by the control information is validation of voice input by the voice input / output unit,
The information processing apparatus according to any one of (6) to (8), wherein the process corresponding to the process indicated by the control information is a process relating to call origination or call arrival of a voice call.
(10)
The process indicated by the control information is invalidation of voice input by the voice input / output unit,
The information processing apparatus according to (9), wherein the process corresponding to the process indicated by the control information is a process related to disconnection of a voice call.
(11)
The control unit cancels the transmission of the message when the process indicated by the control information is a process of changing an environment setting of the information processing apparatus, according to any one of (6) to (8). Information processing equipment.
(12)
A wireless communication unit that performs wireless communication with another information processing apparatus;
Performing processing corresponding to a case where an image is transmitted to the other information processing apparatus via the wireless communication unit and an operation to a designated area of the image displayed by the other information processing apparatus is acquired; A control unit that transmits control information for enabling the control to the other information processing apparatus via the wireless communication unit;
An information processing apparatus comprising:
(13)
The control unit is controlled to execute processing corresponding to processing indicated by the control information based on a message received from the other information processing apparatus via the wireless communication unit. Information processing device.
(14)
The control unit, when the image includes an operable area, the information specifying the area, the information indicating the type of the operation to be acquired in the area, and the operation in the area. (12) or (13), wherein the control information including a combination of information indicating processing to be executed by another information processing apparatus is transmitted to the other information processing apparatus via the wireless communication unit. Information processing device.
(15)
The information according to any one of (12) to (14), wherein the control information is transmitted using an extension of one of messages defined in Wi-Fi CERTIFIED Miracast (registered trademark). Processing unit.
(16)
A transmitting device and a receiving device;
The sender device is
A first wireless communication unit that performs wireless communication with the receiving device;
Performing a process corresponding to a case where an image is transmitted to the reception side device via the first wireless communication unit and an operation to a designated area is detected among the images displayed by the reception side device; A first control unit that transmits control information for enabling the reception side apparatus via the first wireless communication unit;
Have
The receiver device is
A second wireless communication unit that performs wireless communication with the transmission side apparatus;
A display unit for displaying the image received from the transmitting device by the second wireless communication unit;
An acquisition unit configured to acquire an operation on a region indicated by the control information received from the transmission-side device by the second wireless communication unit in the image displayed by the display unit;
A control unit configured to perform a process indicated by the control information according to the operation acquired by the acquisition unit;
An information processing system having
(17)
Wireless communication with another information processing apparatus;
Displaying an image received from the other information processing apparatus;
Acquiring an operation to a region indicated by control information received from the other information processing apparatus among the displayed images;
Controlling the information processing apparatus to perform processing indicated by the control information in accordance with the acquired operation;
An information processing method performed by a processor, including:

Reference Signs List 1 communication system 100 sink device 110 wireless communication unit 120 stream transmission / reception unit 130 compression / decompression unit 140 display unit 150 audio input / output unit 160 control signal transmission / reception unit 170 control unit 180 user interface unit 200 source device 210 wireless communication unit 220 stream transmission / reception unit 230 Compression / expansion unit 240 Content acquisition unit 250 Content reproduction unit 260 Control signal transmission / reception unit 270 Control unit 280 User interface unit 300 Base station 400 External device

Claims (13)

An information processing apparatus,
A wireless communication unit that performs wireless communication with another information processing apparatus;
A display unit for displaying an image received from the other information processing apparatus by the wireless communication unit;
An acquisition unit configured to acquire an operation on a region indicated by control information received from the other information processing apparatus by the wireless communication unit among the images displayed by the display unit;
The information processing apparatus controls the first process indicated by the control information to be performed according to the operation acquired by the acquisition unit, and the other information processing apparatus corresponds to the first process. A control unit configured to control transmission of a message for executing a second process different from the process 1 to the other information processing apparatus via the wireless communication unit;
An audio input / output unit that inputs and outputs audio;
Equipped with
The process indicated by the control information is validation of voice input by the voice input / output unit,
An information processing apparatus , wherein the process corresponding to the process indicated by the control information is a process relating to call origination or call reception of a voice call .   The control information is information for specifying an area where the acquisition unit should acquire an operation, information indicating a type of operation that the acquisition unit should acquire, and the control unit controls when the operation to the area is received. The information processing apparatus according to claim 1, further comprising a combination of information indicating the first process to be performed.   The information processing apparatus according to claim 2, wherein the control information includes information indicating whether the combination is valid or invalid.   The information processing apparatus according to claim 2, wherein the control information includes a plurality of combinations.   The information processing apparatus according to any one of claims 1 to 4, wherein the wireless communication unit bidirectionally establishes a connection by Wi-Fi CERTIFIED MIRACAST (registered trademark) with the information processing apparatus.   The said message contains the operation information which shows the said operation acquired by the said acquisition part, and is transmitted using TCP / IP (Transmission Control Protocol / Internet Protocol) using UIBC (User Input Back Channel). The information processing apparatus as described in any one of -5.   The information processing apparatus according to any one of claims 1 to 6, wherein the message includes a message for PAUSE setting or PLAY setting of Real Time Streaming Protocol (RTSP). The process indicated by the control information is invalidation of voice input by the voice input / output unit,
The information processing apparatus according to any one of claims 1 to 7, wherein the process corresponding to the process indicated by the control information is a process related to disconnection of a voice call. An information processing apparatus,
A wireless communication unit that performs wireless communication with another information processing apparatus;
A first process corresponding to the case where an image is transmitted to the other information processing apparatus via the wireless communication unit and an operation to a designated area of the image displayed by the other information processing apparatus is acquired A control unit that transmits control information for enabling the other information processing apparatus to perform the other information processing apparatus via the wireless communication unit;
Equipped with
The control unit performs second processing different from the first processing corresponding to the first processing based on a message received from the other information processing apparatus via the wireless communication unit. Control to run ,
The other information processing apparatus includes an audio input / output unit that inputs and outputs audio.
The process indicated by the control information is validation of voice input by the voice input / output unit,
An information processing apparatus , wherein the process corresponding to the process indicated by the control information is a process relating to call origination or call reception of a voice call . The control unit, when the image includes an operable area, the information specifying the area, the information indicating the type of the operation to be acquired in the area, and the operation in the area. The information processing apparatus according to claim 9 , wherein the control information including a combination of information indicating processing to be executed by another information processing apparatus is transmitted to the other information processing apparatus via the wireless communication unit. The control information, Wi-Fi CERTIFIED Miracast sent with those extended one defined messages (registered trademark), the information processing apparatus according to claim 9 or 10. A transmitting device and a receiving device;
The sender device is
A first wireless communication unit that performs wireless communication with the receiving device;
A first process corresponding to a case where an image is transmitted to the reception side device via the first wireless communication unit, and an operation to a designated area of the image displayed by the reception side device is detected. A first control unit that transmits control information for enabling the receiving device to perform to the receiving device via the first wireless communication unit;
Have
The receiver device is
A second wireless communication unit that performs wireless communication with the transmission side apparatus;
A display unit for displaying the image received from the transmitting device by the second wireless communication unit;
An acquisition unit configured to acquire an operation on a region indicated by the control information received from the transmission-side device by the second wireless communication unit in the image displayed by the display unit;
According to the operation acquired by the acquisition unit, the reception-side device controls the first process indicated by the control information, and the transmission-side device corresponds to the first process. A second control unit configured to control transmission of a message for executing a second process different from the process of 1 to the transmission side apparatus via the second wireless communication unit;
An audio input / output unit that inputs and outputs audio;
Have
The first control unit controls the transmission side device to execute the second process based on a message received from the reception side device via the first wireless communication unit .
The process indicated by the control information is validation of voice input by the voice input / output unit,
An information processing system , wherein the process corresponding to the process indicated by the control information is a process relating to call origination or call reception of a voice call . Wireless communication with another information processing apparatus;
Displaying an image received from the other information processing apparatus;
Acquiring an operation to a region indicated by control information received from the other information processing apparatus among the displayed images;
The information processing apparatus is controlled to perform a first process indicated by the control information according to the acquired operation, and the first process corresponding to the first process is performed on the other information processing apparatus Controlling to transmit a message for executing a different second process to the other information processing apparatus;
Audio input and output,
Only including,
The process indicated by the control information is activation of voice input,
An information processing method to be executed by a processor , wherein a process corresponding to the process indicated by the control information is a process relating to call origination or call reception of a voice call .

Images