JP2015133637A - Information processing apparatus and operation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus which allows an image and sound output apparatus to perform remote control operation, e.g., sound adjustment, while performing the mirroring operation of the image and sound output apparatus, and to provide an operation system.SOLUTION: A smartphone 1 is configured to display an image for remote control operation input in a display 12, while performing the mirroring operation for displaying a mirroring image in a TV2, by transmitting the mirroring image and mirroring sound via a wireless LAN communication section 14, to perform operation control of the smartphone 1 itself on the basis of the operation information of remote control operation thus received, and to perform operation control of the TV.

Description

  The present invention relates to an information processing device and an operation system, and particularly relates to an information processing device and an operation system including a display unit that displays an image.

  Conventionally, an information processing device and an operation system including a display unit that displays an image are known (for example, see Patent Document 1).

  Patent Document 1 discloses an apparatus (information processing device) including an internal display (display unit) that outputs and displays image data stored in a first frame buffer and a scaler. The apparatus is configured to store image data in a first frame buffer, output the image data in the first frame buffer, and display it on the internal display of the apparatus. The scaler then performs a scaling operation on the image data in the first frame buffer, stores the scaled image data in the second frame buffer, and scales the image data in the second frame buffer. Data is output and displayed on an external display device, and image data (image content) is mirrored.

  Here, in the apparatus described in Patent Document 1, when the application software is operated, it is considered that the image of the operated application software is displayed by performing a mirroring operation on the external display device. . For example, when an application software having a function as a remote control is operated in order to operate a remote control function such as sound adjustment of the external display device during the mirroring operation for the external display device, the remote control operation is performed. It is considered that an image for the display is displayed by mirroring on the external display device.

JP 2012-177902 A

  However, as described above, the device as described in Patent Document 1 has a function as a remote controller in order to operate a remote control function such as sound adjustment of the external display device while performing a mirroring operation on the external display device. When the application software is operated, the image for remote control operation is displayed by mirroring on the external display device, so the user can adjust the audio of the external display device while checking the contents of the image / audio content. There is a problem that remote control operations such as the above cannot be performed.

  The present invention has been made to solve the above-described problems, and one object of the present invention is to perform audio adjustment on an image / audio output device while performing a mirroring operation on the image / audio output device. It is to provide an information processing device and an operation system capable of performing remote control operations such as the above.

  In order to achieve the above object, an information processing apparatus according to a first aspect of the present invention has an information processing function having a function of mirroring a mirroring image and a mirroring sound with respect to an external image / audio output apparatus and a function of a remote controller. A communication unit that communicates with an external audio / video output device, a display unit for displaying a mirroring image and a remote control operation input image different from the mirroring image, and a remote control operation input displayed on the display unit An operation receiving unit that receives an operation by a user on an image, and a control unit that performs remote control operation control based on operation information of a remote control operation received by the operation receiving unit. The control unit mirrors via a communication unit. Image and mirroring sound are transmitted, and the mirroring image is displayed on an external image / audio output device. The remote control operation input image is displayed on the display unit while performing the mirroring operation, and the operation control is performed on the information processing device itself based on the received operation information of the remote control operation. It is comprised so that operation control may be performed with respect to. In general, the mirroring operation means that the control unit of the information processing device stores the mirroring image and the mirroring sound in the first storage unit (buffer) included in the control unit of the information processing device, While displaying the mirroring image stored in the first storage unit on the display unit of the information processing device, the mirroring image and the mirroring sound are stored in the second storage unit included in the control unit of the information processing device. The control for displaying the mirroring image stored in the second storage unit on the display unit of the external video / audio output device. The mirroring operation in the present application refers to the above description and the display unit of the information processing device. The control of a broader concept including the case where the image for mirroring memorize | stored in 1 memory | storage part is not displayed is represented.

  In the information processing device according to the first aspect of the present invention, as described above, the remote control operation input image is displayed on the display unit while performing the mirroring operation of displaying the mirroring image on the external image / audio output device, The user controls the information processing device itself based on the received operation information of the remote control operation, and also performs the operation control on the external image / audio output device. While confirming the output mirroring image and mirroring sound, the contents of the mirroring image and mirroring sound can be operated, and the remote control operation for controlling the operation of the image / sound output device is performed. be able to. As a result, even if the remote control device attached to the external audio / video output device is not changed from the information processing device, the image / audio output device can be adjusted for audio while performing the mirroring operation. Remote control operation can be performed.

  In the information processing device according to the first aspect, preferably, the control unit can display a remote control operation input image on the display unit during the non-mirroring operation, as in the mirroring operation. Is configured to perform operation control on the information processing device itself and to perform operation control on an external video / audio output device. With this configuration, the user can control the operation of the information processing device itself and the external video / audio output device by the same operation device (information processing device) and operation method regardless of the non-mirroring operation or the mirroring operation. Operation control can be performed.

  In the information processing device according to the first aspect, preferably, the operation reception unit includes a touch panel, and the touch panel is configured to receive a touch operation by a user in the remote control operation input image displayed on the display unit. . If comprised in this way, since the touch panel can receive operation according to a user's finger | toe movement, a user can perform remote control operation input easily.

  In this case, preferably, the operation accepting unit is configured such that the control unit detects the touch operation as a hovering operation that is an aerial operation on the touch panel surface of the touch panel when the touch panel detects the touch operation. And generating two-dimensional position coordinate information as a touch operation on the touch panel surface of the touch panel when a predetermined operation is received after the touch operation is detected, and 3 A hovering cursor indicating a position coordinate corresponding to at least the three-dimensional position coordinate information of the two-dimensional position coordinate information and the two-dimensional position coordinate information is superimposed on the mirroring image and displayed on an external video / audio output device. It is configured to perform control. If comprised in this way, even if it is a case where a touch panel does not have the performance which detects the user's finger | toe in the air on a touch panel surface, a hovering operation can be received. As a result, since the user can visually recognize the hovering cursor indicating the position coordinates corresponding to the hovering operation displayed on the external image / audio output device, the user does not need to visually recognize the display unit of the information processing device. Operation control can be performed on an external video / audio output device.

  In the information processing device according to the first aspect described above, preferably, the operation reception unit includes a touch panel, and the touch panel is an aerial image on the touch panel surface of the touch panel by the user in the remote control operation input image displayed on the display unit. The control unit is configured to receive a hovering operation that is an operation, and the control unit performs control for superimposing a hovering cursor indicating a position coordinate corresponding to the hovering operation on an image for mirroring to be displayed on an external video / audio output device. Configured to do. If comprised in this way, a user can visually recognize the hovering cursor which shows the position coordinate corresponding to the hovering operation displayed on the external audio-video output apparatus. As a result, the user can perform operation control on the external video / audio output device without visually recognizing the display unit of the information processing device.

  In the information processing device according to the first aspect described above, preferably, the control unit transmits the mirroring image and the mirroring sound via the communication unit, and causes the external image / sound output device to display the mirroring image. While executing the first application software that performs the operation, the remote control operation input image is displayed on the display unit, and the operation control is performed on the information processing device itself based on the received operation information of the remote control operation. Second application software for performing operation control on the image / sound output device is configured to be executed. With this configuration, the first application software is executed and the second application software is executed in parallel while performing the mirroring operation, so that the operation control for the information processing device itself and the operation control for the external video / audio output device are performed. It can be performed.

  In this case, preferably, the first application software and the second application software are configured by a single application software integrated with each other. If comprised in this way, since it is not necessary to operate two application software simultaneously, the process which a control part performs can be simplified.

  In the information processing apparatus configured to execute the second application software while executing the first application software, preferably, the control unit displays the mirroring image while displaying the image for mirroring on the external image / audio output apparatus. Instead of displaying the mirroring image on the part, the remote control operation input image is displayed and the operation by the user is accepted. With this configuration, since the mirroring image is not displayed on the display unit of the information processing device, the user can easily display the remote control operation input image displayed on the display unit of the information processing device during remote control operation. It can be visually recognized and operability can be improved.

  An operation system according to a second aspect of the present invention includes an image / sound output device, and an information processing device having a function of mirroring a mirroring image and a mirroring sound with respect to the image / sound output device and a function as a remote control, An image / audio output device includes an image / audio output device side communication unit that communicates with an information processing device, an image / audio output device side display unit that displays a mirroring image, and an image / audio output device side audio output unit that outputs mirroring sound. The information processing device includes an information processing device side communication unit that communicates with the image and sound output device, an information processing device side display unit that displays an image for remote control operation input that is different from the mirroring image and the mirroring image, Information for accepting a user operation on the remote control operation input image displayed on the information processing device display A processing device side operation receiving unit, and an information processing device side control unit that performs remote control operation control based on operation information of the remote control operation received by the information processing device side operation receiving unit. The mirroring image and the mirroring sound are transmitted to the image / sound output device via the information processing device side communication unit and the image / sound output device side communication unit, and the mirroring image is displayed on the image / sound output device side display unit. While performing a mirroring operation, the remote control operation input image is displayed on the information processing device side display unit, and based on the received remote control operation information, the information processing device itself is controlled and video and audio are output. It is comprised so that operation control may be performed with respect to the apparatus side audio | voice output part.

  In the operation system according to the second aspect of the present invention, as described above, the remote control operation input image is displayed on the information processing device-side display unit while performing the mirroring operation for displaying the mirroring image on the image / sound output device-side display unit. Is displayed, and the user performs an operation control on the information processing device itself based on the received operation information of the remote control operation, and also performs an operation control on the audio output unit on the image audio output device, thereby allowing the user to While confirming the mirroring image and the mirroring sound output to the audio output device side display unit, the contents of the mirroring image and the mirroring audio can be operated, and the image audio output device side audio output unit Remote control operation for performing operation control can be performed. As a result, even if the remote control device attached to the audio / video output device is not changed from the information processing device, the remote control operation such as audio adjustment is performed on the audio / video output device while performing the mirroring operation on the audio / video output device. It can be performed.

  An operation system according to a third aspect of the present invention mirrors an image and sound output device, a first information processing device having a remote control function for the image and sound output device, and a mirroring image and a mirroring sound for the image and sound output device. The image / audio output device includes a second information processing device having a function to perform communication. The image / audio output device communicates with the first information processing device and the second information processing device, and the image / audio displays a mirroring image. The first information processing device includes an output device side display unit and an image / sound output device side sound output unit that outputs mirroring sound, and the first information processing device communicates with the image / sound output device and the second information processing device. Based on the operation information of the remote control operation received by the side communication unit, the remote control operation reception unit that receives the operation by the user, and the remote control operation reception unit A first information processing device side control unit that performs remote control operation control, and the second information processing device includes a second information processing device side communication unit that communicates with the audio / video output device and the first information processing device, and mirroring. The image for mirroring and the sound for mirroring are transmitted to the image / sound output device via the second information processing device side display unit for displaying the image, the second information processing device side communication unit and the image / sound output device side communication unit. And a second information processing device side control unit configured to perform a mirroring operation for displaying a mirroring image on the image / audio output device side display unit. When the information processing device and the image / sound output device are performing the mirroring operation, based on the operation information of the remote control operation received by the remote control operation reception unit of the first information processing device. , It performs operation control for the second information processing apparatus is configured to perform operation control with respect to the image and sound output device-side audio output unit.

  In the operation system according to the third aspect of the present invention, as described above, when the second information processing device and the video / audio output device perform the mirroring operation, Based on the operation information of the remote control operation received by the remote control operation receiving unit, the operation control is performed on the second information processing device, and the operation control is performed on the audio output unit on the audio / video output device side. Thus, the user confirms the mirroring image and the mirroring sound output to the display unit on the image / sound output device side, while the contents of the mirroring image and the mirroring sound are the same as those in the normal remote control operation. The remote control can be operated using the information processing device, and the remote control for performing the operation control on the audio output unit on the image / audio output device side It can be performed using the first information processing equipment down operation in normal remote procedure analogous.

  According to the present invention, as described above, an information processing device and an operation system capable of performing remote control operations such as sound adjustment on an image / audio output device while performing a mirroring operation on the image / audio output device. Can be provided.

It is a block diagram showing the whole operation system composition by a 1st embodiment of the present invention. It is a figure for demonstrating the image display at the time of the mirroring operation | movement by the operation system by 1st Embodiment of this invention. It is a figure for demonstrating the image display at the time of the non-mirroring operation | movement by the operation system by 1st Embodiment of this invention. It is a figure for demonstrating the image generation by the operation system by 1st Embodiment of this invention. It is a flowchart (1) for demonstrating the operation processing of the operation system by 1st Embodiment of this invention. It is a flowchart (2) for demonstrating the operation processing of the operation system by 1st Embodiment of this invention. It is a figure for demonstrating the image generation by the operation system by 2nd Embodiment of this invention. It is a figure for demonstrating the image display at the time of the mirroring operation | movement by the operation system by 2nd Embodiment of this invention. It is a flowchart for demonstrating the operation processing of the operation system by 2nd Embodiment of this invention. It is the block diagram which showed the whole structure of the operation system by 3rd Embodiment of this invention. It is a figure for demonstrating the image display at the time of the mirroring operation | movement by the operation system by 3rd Embodiment of this invention. It is a figure for demonstrating the image display at the time of the non-mirroring operation | movement by the operation system by 3rd Embodiment of this invention. It is a figure for demonstrating the image generation by the operation system by 3rd Embodiment of this invention. It is a flowchart (1) for demonstrating the operation processing of the operation system by 3rd Embodiment of this invention. It is a flowchart (2) for demonstrating the operation processing of the operation system by 3rd Embodiment of this invention. It is a figure (1) for demonstrating the hovering cursor display at the time of the mirroring operation | movement by the operation system by 4th Embodiment of this invention. It is a figure (2) for demonstrating the hovering cursor display at the time of the mirroring operation | movement by the operation system by 4th Embodiment of this invention. It is a figure (1) for demonstrating the hovering cursor display at the time of the mirroring operation | movement by the operation system by 5th Embodiment of this invention. It is a figure (2) for demonstrating the hovering cursor display at the time of the mirroring operation | movement by the operation system by 5th Embodiment of this invention.

  DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments embodying the present invention will be described below with reference to the drawings.

(First embodiment)
With reference to FIGS. 1-4, the structure of the operation system 100 by 1st Embodiment of this invention is demonstrated.

  The operation system 100 according to the first embodiment of the present invention includes a smartphone 1 and a TV 2 as shown in FIG. The smartphone 1 is an example of the “information processing device” in the present invention. The TV 2 is an example of the “image / sound output device” in the present invention.

  As shown in FIG. 1, the smartphone 1 includes an SOC (System On a chip) 11, a display unit 12, a touch panel 13, a wireless LAN (Local Area Network) communication unit 14, a voice control circuit 15, And a speaker 16. The SOC 11 is an example of the “control unit” in the present invention. The touch panel 13 is an example of an “operation receiving unit”. The wireless LAN communication unit 14 is an example of a “communication unit”.

  As shown in FIG. 1, the TV 2 communicates with the CPU 21, the display unit 22, the speaker 23, the display control circuit 24, the audio control circuit 25, the wireless LAN communication unit 26, the broadcast receiving unit 27, and the communication. An I / F 28 and a reception I / F 29 are provided. The speaker 23 and the sound control circuit 25 are an example of the “image / sound output device side sound output unit” of the present invention. The wireless LAN communication unit 26 is an example of the “video / audio output device side communication unit” of the present invention.

  As shown in FIG. 1, the display unit 22 of the TV 2 is configured with a liquid crystal panel or the like, and is configured to be able to display image data output from the CPU 21 via a display control circuit 24. The speaker 23 is configured to be capable of outputting sound data output from the CPU 30 via the sound control circuit 25. The wireless LAN communication unit 26 is configured to be able to communicate with the smartphone 1. The broadcast receiving unit 27 is configured to receive a TV broadcast image signal and a TV broadcast audio signal. The communication I / F 28 is configured to connect the wireless LAN communication unit 26 and the CPU 21. The reception I / F 29 is configured to connect the broadcast receiving unit 27 and the CPU 21.

  The CPU 21 of the TV 2 is configured to acquire the mirroring image data and the mirroring audio data from the smartphone 1 via the wireless LAN communication unit 26 and the communication I / F 28, and displays the acquired mirroring image. In addition to being displayed on the unit 22, the acquired mirroring sound is output to the speaker 23. Further, the CPU 21 is configured to perform operation control based on operation information of remote control operation.

  And as shown in FIG. 1, the display part 12 of the smart phone 1 is comprised by the liquid crystal panel etc., and is comprised so that the image output from SOC11 may be displayed. The touch panel 13 includes a transparent projected capacitive touch panel, and is configured integrally with the display unit 12. Accordingly, the touch panel 13 is configured to be able to accept a touch operation corresponding to the position of the operation unit image displayed on the display unit 12. The touch operation is configured to be performed using a touch pen, a user's finger, or the like. The sound control circuit 15 is configured to convert the sound signal output from the SOC 11 into a signal for causing the speaker 16 to output sound. For example, the speaker 16 is configured to output a call voice during a call with another smartphone, and output a sound of the image / audio content when reproducing the image / audio content. The wireless LAN communication unit 14 is configured to be able to communicate with the TV 2 by wireless communication.

  As shown in FIG. 1, the SOC 11 of the smartphone 1 includes a CPU (Central Processing Unit) 30, an image generation unit 31, a RAM (Random Access Memory) 32, a flash memory 33, a coordinate information generation unit 34, and the like. A display control circuit 35, a touch panel I / F (interface) 36, a communication I / F 37, a voice I / F 38, and an internal bus 39.

  As shown in FIG. 1, the components in the SOC 11 of the smartphone 1 are configured to be connectable by an internal bus 39. The touch panel I / F 36 is configured to connect the touch panel 13 and the SOC 11. The communication I / F 37 is configured to connect the wireless LAN communication unit 14 and the SOC 11. The audio I / F 38 is configured to connect the audio control circuit 15 and the SOC 11.

  As shown in FIG. 1, the display control circuit 35 is a circuit that performs control for causing the display unit 12 to display an image. For example, the display control circuit 35 displays an image on the display unit 12 by controlling the orientation of the liquid crystal molecules of the liquid crystal panel of the display unit 12 based on the image signal.

  The flash memory 33 of the smartphone 1 transmits the image for mirroring and the sound for mirroring to the flash memory 33 via the wireless LAN communication unit 14 and the software constituting the operating system, and sends the image for mirroring and the sound for mirroring to the TV 2. Based on the program that constitutes the image / sound application software (hereinafter referred to as the image / sound application) that outputs and performs the mirroring operation, and the remote control operation input image is displayed on the display unit 12, and based on the received operation information of the remote control operation, A program that configures remote control application software (hereinafter referred to as a remote control application) that performs operation control on the smartphone 1 and also performs operation control on the TV 2 is stored (installed) in advance. The flash memory 33 stores image / audio data including a mirroring image, a mirroring sound, and a remote control operation input image. And CPU30 of the smart phone 1 is comprised so that the process mentioned later may be performed by reading the operating system software and application software which are memorize | stored in the flash memory 33, and executing it. The image / sound application is an example of the “first application software” in the present invention. The remote control application is an example of the “second application software” in the present invention.

  Moreover, in 1st Embodiment, as shown in FIG. 2 and FIG. 4, CPU30 of the smart phone 1 performs the mirroring operation | movement which displays the image for mirroring on TV2 by performing an image audio | voice application, smart phone 1 itself It is comprised so that operation control may be performed. Specifically, when the image / audio application performs a mirroring operation to output a mirroring image and a mirroring sound to the display unit 12 of the smartphone 1 and the display unit 22 of the TV 2, the touch panel 13 of the smartphone 1 receives the operation by the user. On the basis of the operation information, the selection of the type of image and sound, the start and stop of reproduction, and the like can be operated.

  Moreover, in 1st Embodiment, as shown in FIG. 2 and FIG. 4, CPU30 of the smart phone 1 displays the image for remote control operation input on the display part 12 of the smart phone 1, by executing a remote control application, and the touch panel 13 is shown. Is configured to perform audio adjustment control on the TV 2 based on the operation information of the remote control operation accepted by the TV 2. For example, when the touch panel 13 accepts a user's touch operation on the “volume up” button (reference A in FIG. 2) that is a display unit image displayed on the display unit 12 of the smartphone 1, the CPU 30 outputs the audio of the TV 2. The remote control operation information signal for performing control to increase is configured to be transmitted via the wireless LAN communication unit 14. Then, the CPU 21 of the TV 2 acquires a remote control operation information signal for performing control for increasing the audio output of the TV 2 via the wireless LAN communication unit 26. The CPU 21 of the TV 2 is configured to increase the sound output from the speaker 23 by increasing the amplification of the sound signal of the sound control circuit 25 of the TV 2 based on the acquired remote control operation information signal. The sound adjustment control is an example of the “operation control” in the present invention.

  As shown in FIG. 1, the RAM 32 of the smartphone 1 includes a first frame buffer 32 a, a second frame buffer 32 b, and a third frame buffer 32 c, and the display unit 12 of the smartphone 1 by the image generation unit 31. Alternatively, the image data converted into a format (resolution or the like) suitable for the display unit 22 of the TV 2 can be recorded.

  Here, in 1st Embodiment, as shown in FIG.2 and FIG.4, when the smart phone 1 and TV2 are performing mirroring operation | movement, CPU30 of the smart phone 1 is set to the display part 12 of the smart phone 1, It is configured to display the remote control operation input image without displaying the mirroring image (by executing the mirroring operation in the background), and the CPU 21 of the TV 2 displays the mirroring image on the display unit 22 of the TV 2. Is configured to be displayed.

  Specifically, as illustrated in FIG. 4A, the image generation unit 31 converts the mirroring image stored in the flash memory 33 into mirroring image data suitable for display on the display unit 12 of the smartphone 1. (Scaling) and recording in the first frame buffer 32a. Then, the image generation unit 31 converts the mirroring image data suitable for display on the display unit 12 of the smartphone 1 stored in the first frame buffer 32a into mirroring image data suitable for display on the display unit 22 of the TV 2. It is configured to convert and store it in the second frame buffer 32b. Then, the CPU 30 of the smartphone 2 is configured to transmit the mirroring image data suitable for display on the display unit 22 of the TV 2 stored in the second frame buffer 32 b to the TV 2 via the wireless LAN communication unit 14. ing.

  Here, in the first embodiment, the image generation unit 31 converts the remote control operation input image stored in the flash memory 33 into remote control operation input image data suitable for display on the display unit 12 of the smartphone 1. The remote control operation input image suitable for display on the display unit 12 of the smartphone 1 stored in the third frame buffer 32c and displayed in the third frame buffer 32c is displayed on the display unit 12 of the smartphone 1. Has been.

  Further, when the CPU 30 of the smartphone 1 receives a user touch operation on the operation unit image on the remote control operation input image displayed on the display unit 12 of the smartphone 1, the CPU 30 does not perform a mirroring operation described later. Similar to the case where the remote control application is executed, a predetermined operation of the operation unit image is performed. For example, when the display of “high volume” (symbol A in FIG. 2) is operated, the CPU 30 of the smartphone 1 outputs a remote control operation information signal for increasing the audio output of the TV 2 via the wireless LAN communication unit 14. Configured to send. The CPU 21 of the TV 2 acquires remote control operation information via the wireless LAN communication unit 26, controls the audio control circuit 25 based on the acquired operation information of the remote control operation, and the level of audio output from the speaker 23 Is configured to increase.

  In the first embodiment, as shown in FIGS. 3 and 4, when the smartphone 1 and the TV 2 are in a non-mirroring operation, the CPU 30 of the smartphone 1 displays an image for remote control operation input on the display unit 12 of the smartphone 1. The CPU 21 of the TV 2 is configured to display the TV broadcast image received from the broadcast receiving unit 27 on the display unit 22 of the TV 2.

  Specifically, as illustrated in FIG. 4B, the image generation unit 31 of the smartphone 1 is suitable for displaying the remote control operation input image stored in the flash memory 33 on the display unit 12 of the smartphone 1. The remote control operation input image data is converted into remote control operation input image data and stored in the third frame buffer 32c, and the remote control operation input image suitable for display on the display unit 12 of the smartphone 1 stored in the third frame buffer 32c is displayed on the smartphone 1. It is configured to display on the display unit 12. On the other hand, as shown in FIGS. 1 and 3, the CPU 21 of the TV 2 acquires the TV broadcast image signal and the TV broadcast audio signal received from the broadcast receiving unit 27 via the reception I / F 29, and displays them on the display unit 22. A broadcast image is displayed and broadcast audio is output to the speaker 23.

  Next, an operation processing flow of the operation system 100 according to the first embodiment will be described with reference to FIGS. 5 and 6. Processing in the smartphone 1 is performed by the CPU 30. The processing in the TV 2 is performed by the CPU 21.

  First, as shown in FIG. 5, in the smartphone 1, in step S <b> 1, it is determined whether or not a user input operation for starting the image / sound application is accepted. This determination is repeated until an input operation for starting the image / sound application is received. When the input operation is received, the process proceeds to step S2.

  Next, in step S2, the image / audio application stored in the flash memory 33 is read, and the image / audio application is activated. Thereafter, the process proceeds to step S3. In step S3, the image / sound data stored in the flash memory 33 is read. Thereafter, the process proceeds to step S4. In step S4, the read audio data is stored in the RAM 32. Thereafter, the process proceeds to step S5.

  Next, in step S5, the image data is stored in the first frame buffer 32a. The image generation unit 31 converts the read image data into a format suitable for the display unit 12 of the smartphone 1 and stores it in the first frame buffer 32a. Then, it progresses to step S6. In step S6, the image data is stored in the second frame buffer 32b. The image generation unit 31 converts the read image data into a format suitable for the display unit 22 of the TV 2 and stores it in the second frame buffer 32b. Then, it progresses to step S7.

  In step S7, the image data (mirroring image data) stored in the second frame buffer 32b and the audio data (mirroring audio data) stored in the RAM 32 are transmitted to the TV 2. Thereafter, the process proceeds to step S8.

  In step S8, the remote control application stored in the flash memory 33 is read, and the remote control application is activated. Thereafter, the process proceeds to step S9.

  In step S9, the remote control operation input image data stored in the flash memory 33 is read out. Then, it progresses to step S10. In step S10, the read image data for remote control operation input is stored in the third frame buffer 32c. Then, it progresses to step S11 (refer FIG. 6).

  On the other hand, in the TV 2, it is determined in step S21 whether image / audio data has been received. If image / audio data has been received, the process proceeds to step S22. If image / audio data is not acquired, the process proceeds to step S23 (see FIG. 6).

  In step S22, the acquired mirroring image is displayed on the display unit 22 of the TV 2, and the acquired mirroring sound is output from the TV speaker 23. Then, it progresses to step S23 (refer FIG. 6).

  As shown in FIG. 6, in the smartphone 1, the remote control operation input image stored in the third frame buffer 32 c is displayed on the display unit 12 of the smartphone 1 in step S <b> 11. Here, in the first embodiment, the image data (mirroring image data) stored in the second frame buffer 32b performed in step S7 and the audio data (mirroring audio data) stored in the RAM 32 to the TV 2 are stored. Transmission continues. Then, the process proceeds to step S12.

  Then, in step S12, it is determined whether or not a user's remote control operation input has been accepted. This determination is repeated until a remote control operation input is received. If a remote control operation input is received, the process proceeds to step S13.

  In step S13, a remote control operation information signal for performing operation control of the TV 2 is transmitted. Then, it returns to step S12.

  On the other hand, as shown in FIG. 6, in the TV 2, in step S <b> 23, it is determined whether or not the remote control operation information signal transmitted from the smartphone 1 has been acquired. If the remote control operation information signal is acquired, the process proceeds to step S24. If the remote control operation information signal is not acquired, the process returns to step S21 (see FIG. 5).

  In step S24, the audio output from the speaker 23 is adjusted based on the acquired remote control operation input signal. Then, it returns to step S21 (refer FIG. 5).

  In the first embodiment, the following effects can be obtained.

  In the first embodiment, as described above, the SOC 11 of the smartphone 1 performs the mirroring operation of transmitting the mirroring image and the mirroring sound via the wireless LAN communication unit 14 and displaying the mirroring image on the TV 2. On the other hand, the display unit 12 is configured to display an image for remote control operation input and to perform operation control on the smartphone 1 itself and operation control on the TV 2 based on the received operation information of the remote control operation. Accordingly, the user can operate the contents of the mirroring image and the mirroring sound while confirming the mirroring image and the mirroring sound output to the TV 2 and can also control the TV 2 for operation control. The operation can be performed. As a result, it is possible to perform operations such as audio adjustment on the TV 2 while performing a mirroring operation on the TV 2 without switching from the smartphone 1 to the remote control device attached to the TV 2.

  In the first embodiment, as described above, the SOC 11 of the smartphone 1 can display the remote control operation input image on the display unit 12 during the non-mirroring operation, as in the mirroring operation. Based on the operation information, the operation control for the smartphone 1 itself is performed, and the operation control for the TV 2 is performed, so that the user can operate the same operation device (smartphone regardless of the non-mirroring operation or the mirroring operation). According to 1) and the operation method, the operation control for the smartphone 1 itself and the operation control for the TV 2 can be performed.

  Moreover, in 1st Embodiment, as mentioned above, the touch panel 13 of the smart phone 1 is comprised so that the touch operation by a user may be received in the image for remote control operation input displayed on the display part 12. FIG. Thereby, the touch panel 13 can receive an operation in accordance with the movement of the user's finger. As a result, the user can easily perform remote control operation input.

  In the first embodiment, as described above, the mirroring operation in which the SOC 11 of the smartphone 1 transmits the mirroring image and the mirroring sound via the wireless LAN communication unit 14 to display the mirroring image on the TV 2. The remote control operation input image is displayed on the display unit 12 while executing the image / sound application for performing the operation control for the smartphone 1 and the operation control for the TV 2 based on the received operation information of the remote control operation. Configure to run the remote control app. As a result, it is possible to execute the image / audio application and execute the remote control application in parallel with the mirroring operation, thereby performing the operation control on the smartphone 1 itself and the operation control on the TV 2.

  Further, in the first embodiment, as described above, the SOC 11 of the smartphone 1 displays the mirroring image on the TV 2 and displays the remote control operation input image on the display unit 12 without displaying the mirroring image. And configured to accept user operations. Thereby, since the image for mirroring is not displayed on the display part 12 of the smart phone 1, the user can easily visually recognize the remote control operation input image displayed on the display part 12 of the smart phone 1 during the remote control operation. And operability can be improved.

(Second Embodiment)
Next, the configuration of the operation system 101 according to the second embodiment will be described with reference to FIG. 1, FIG. 7, and FIG. Unlike the operation system 100 according to the first embodiment, in which the image / audio application and the remote control application are configured as different application software, the image / audio application and the remote control application are integrated with each other as a single application software (with a remote control function). Image / audio application).

  As shown in FIG. 1, the operation system 101 according to the second embodiment includes a smartphone 5 and a TV 2. The smartphone 5 includes an SOC 51, and the SOC 51 includes a CPU 52, a flash memory 53 in which an image / audio application with a remote control function is stored (installed) in advance, and an image generation unit 54.

  Here, in 2nd Embodiment, as shown in FIG.7 and FIG.8, when the smart phone 5 and TV2 are carrying out mirroring operation | movement (when the image audio | voice application with a remote control function is performed), the smart phone 5 is carried out. The CPU 52 and the CPU 21 of the TV 2 are configured to display a mirroring image on the display unit 12 of the smartphone 5 and the display unit 22 of the TV 2. The display unit 12 of the smartphone 5 is configured to display a remote control operation input image in addition to the mirroring image.

  Specifically, as illustrated in FIG. 7A, the image generation unit 54 is suitable for displaying the mirroring image and the remote control operation input image stored in the flash memory 53 on the display unit 12 of the smartphone 5. The image data is converted into image data and recorded in the first frame buffer 32a. The image generation unit 54 converts the image data suitable for display on the display unit 12 of the smartphone 1 stored in the first frame buffer 32a into image data for mirroring suitable for the display unit 22 of the TV 2, and It is configured to store in the second frame buffer 32b.

  Here, in the second embodiment, as shown in FIG. 8, the image / audio application with a remote control function converts the image data for conversion so that the remote control operation input image is not displayed on the display unit 22 of the TV 2. It is configured to (scaling). Further, when the smartphone 5 and the TV 2 are in the non-mirroring operation, the remote controller operation input image is displayed on the display unit 12 of the smartphone 5 as in the non-mirroring operation of the operation system according to the first embodiment (see FIG. 3). Then, the TV broadcast image is displayed on the display unit 22 of the TV 2. The other configuration of the operation system 101 according to the second embodiment is the same as that of the operation system 100 according to the first embodiment.

  Next, an operation processing flow of the operation system 101 according to the second embodiment will be described with reference to FIG. Processing in the smartphone 5 is performed by the CPU 52. The processing in the TV 2 is performed by the CPU 21.

  First, as illustrated in FIG. 9, in the smartphone 5, in step S <b> 31, it is determined whether or not a user input operation for starting an image / audio application with a remote control function has been received. This determination is repeated until an input operation for starting an image / audio application with a remote control function is received. If an input operation is received, the process proceeds to step S32.

  Next, in step S32, the image / audio application with remote control function stored in the flash memory 53 is read, and the image / audio application with remote control function is activated. Thereafter, the process proceeds to step S33, and the image / sound data stored in the flash memory 53 is read in step S33. Thereafter, the process proceeds to step S34. In step S34, the read audio data is stored in the RAM 32. Thereafter, the process proceeds to step S35.

  Next, in step S35, the image data is stored in the first frame buffer 32a. The image generation unit 54 converts the read image data into a format suitable for the display unit 12 of the smartphone 5 and stores it in the first frame buffer 32a. Thereafter, the process proceeds to step S36. In step S36, the image data is stored in the second frame buffer 32b. The image generation unit 54 converts the read image data into a format suitable for the display unit 22 of the TV 2 and stores it in the second frame buffer 32b. Thereafter, the process proceeds to step S37.

  In step S37, the image stored in the first frame buffer 32a is displayed on the display unit 12 of the smartphone 5, and the sound stored in the RAM 32 is output from the speaker 16 of the smartphone 5. Thereafter, the process proceeds to step S38. In step S38, the image data stored in the second frame buffer 32b and the audio data stored in the RAM 32 are transmitted to the TV 2. Thereafter, the process proceeds to step S39.

  Then, in step S39, it is determined whether or not a user remote control operation input for remote control operation of the TV 2 has been accepted. If accepted, the process proceeds to step S40. If not accepted, the process returns to step S37.

  Next, in step S40, a remote control operation information signal for performing operation control of the TV 2 is transmitted. Thereafter, the process returns to step S37.

  On the other hand, as shown in FIG. 9, in steps S <b> 21 to S <b> 24 in the TV 2, the processes of steps S <b> 21 to S <b> 24 that are the same processes as in the first embodiment are performed.

  In the second embodiment, the following effects can be obtained.

  In the second embodiment, as described above, the video / audio application and the remote control application are configured by a single application software (video / audio application with a remote control function) integrated with each other. As a result, the processing performed by the SOC 51 can be simplified because it is not necessary to operate the two application software simultaneously. Other effects of the operation system 101 according to the second embodiment are the same as those of the operation system 100 according to the first embodiment.

(Third embodiment)
Next, the configuration of the operation system 102 according to the third embodiment will be described with reference to FIGS. In the third embodiment, the operation system 102 further includes a remote controller 7 including a touch panel 73 and a display unit 74, and when the user operates the remote controller 7, the smartphone 6 and the TV 8 perform a mirroring operation, and the smartphone 6 The operation control for the TV 8 and the operation control for the TV 8 are performed. The remote controller 7 is an example of the “first information processing device” in the present invention.

  As shown in FIG. 10, the operation system 102 according to the third embodiment includes a smartphone 6, a remote controller 7, and a TV 8. The smartphone 6 includes an SOC 61, and the SOC 61 includes a CPU 62 and an image generation unit 63. The smartphone 6 includes a wireless LAN communication unit 64 that can communicate with the remote controller 7. The remote controller 7 includes a SOC 71, a wireless LAN communication unit 72 that can communicate with the smartphone 6 and the TV 8, and a touch panel that receives a touch operation from the user. 73 and a display unit 74. The TV 8 includes a CPU 81 and a wireless LAN communication unit 82 that can communicate with the smartphone 6 and the remote controller 7. The SOC 71 is an example of the “first information processing device side control unit” in the present invention. The wireless LAN communication unit 72 is an example of the “first information processing device side communication unit” in the present invention. The touch panel 73 is an example of the “remote control operation accepting unit” in the present invention.

  Then, the SOC 71 of the remote controller 7 is configured to acquire operation information of the touch operation by the user received by the touch panel 73 and transmit the operation information to the smartphone 6 and the TV 8 via the wireless LAN communication unit 64 and the wireless LAN communication unit 72. Has been. That is, when the smartphone 6 and the TV 8 are in a mirroring operation, the remote controller 7 is configured to be operable with respect to the smartphone 6 and the TV 8. When the smartphone 6 and the TV 8 are in a non-mirroring operation, the wireless LAN communication unit 72 and the wireless By transmitting a remote control operation information signal via the LAN communication unit 82, the TV 8 can be operated by a remote control.

  Here, in 3rd Embodiment, as shown in FIG. 11, when the smart phone 6 and TV8 are carrying out mirroring operation | movement, CPU62 of the smart phone 6 and CPU81 of TV8 are the display part 12 of smart phone 6, and TV8. The display unit 22 is configured to display a mirroring image. On the other hand, the SOC 71 of the remote controller 7 is configured to display a remote control operation input image on the display unit 74 of the remote controller 7.

  Specifically, as illustrated in FIGS. 11 and 13, the image generation unit 63 of the smartphone 6 converts the mirroring image stored in the flash memory 33 into a mirroring image suitable for display on the display unit 12 of the smartphone 6. The data is converted into data and recorded in the first frame buffer 32a. Then, the image generation unit 63 converts the mirroring image data suitable for display on the display unit 12 of the smartphone 6 stored in the first frame buffer 32a into mirroring image data suitable for display on the display unit 22 of the TV 8. It is configured to convert and store it in the second frame buffer 32b. The CPU 62 of the smartphone 6 is configured to transmit mirroring image data suitable for display on the display unit 22 to the TV 8 stored in the second frame buffer 32b via the wireless LAN communication unit 64. . The CPU 81 of the TV 8 is configured to display a mirroring image on the display unit 22.

  In the third embodiment, as shown in FIGS. 11 and 13A, the image generation unit in the SOC 71 of the remote controller 7 uses the remote control operation input image stored in the flash memory in the SOC 71 as the remote control. 7 is converted into image data for remote control operation input suitable for display on the display unit 74 and stored in a frame buffer in the SOC 71. The SOC 71 of the remote controller 7 is configured to display a remote control operation input image suitable for display on the display unit 74 of the remote controller 7 stored in the frame buffer in the SOC 71.

  Also, as shown in FIGS. 12 and 13B, when the smartphone 6 and the TV 8 are not in a mirroring operation (during a non-mirroring operation), the CPU 81 of the TV 8 performs TV broadcasting on the display unit 22 of the TV 8. It is configured to display an image. On the other hand, the SOC 71 of the remote controller 7 is configured to display a remote control operation input image on the display unit 74 of the remote controller 7. Other configurations of the operation system 102 according to the third embodiment are the same as those of the operation system 100 according to the first embodiment.

  Next, an operation processing flow of the operation system 102 according to the third embodiment will be described with reference to FIGS. 14 and 15. Processing in the smartphone 6 is performed by the CPU 62. Further, the processing in the TV 8 is performed by the CPU 81. Processing in the remote controller 7 is performed by the SOC 71.

  First, as shown in FIG. 14, steps S1 to S7 in the smartphone 6 and steps S21 and S22 of the TV 8 are performed in steps S1 to S7 in the smartphone 1 and the step S21 of the TV 2 in the first embodiment. This is the same as the processing in step S22. After step S7, the process proceeds to step S51 (see FIG. 15).

  As shown in FIG. 15, in step S <b> 51, the smartphone 6 transmits a remote control application activation signal for activating the remote control application to the remote control 7. Thereafter, the process proceeds to step S52. In step S52, it is determined whether or not a remote control operation input signal has been acquired from the remote controller 7. This determination is repeated until a remote control operation input is acquired. If a remote control operation input is acquired, the process proceeds to step S53.

  In step S53, operation control based on the remote control operation information signal (operation content) is executed, and the process returns to step S52.

  On the other hand, as shown in FIG. 15, in the remote controller 7, in step S <b> 61, it is determined whether or not the remote controller application activation signal transmitted from the smartphone 6 has been received. This determination is repeated until the remote control application activation signal is received. When the remote control application activation signal is received, the process proceeds to step S62. In step S62, the remote control operation input image data stored in the flash memory in the SOC 71 is read, and the remote control application is activated. Then, the process proceeds to step S63.

  In step S 63, the read remote control operation input image is displayed on the display unit 74 of the remote controller 7. Thereafter, the process proceeds to step S64. In step S64, it is determined whether or not a remote control operation input has been accepted. This determination is repeated until a remote control operation input is received. If a remote control operation input is received, the process proceeds to step S65. In step S65, the remote control operation information signal is transmitted to the smartphone 6 and the TV 8. Thereafter, the process returns to step S64.

  Further, as shown in FIG. 15, the TV 2 performs the processes of step S23 and step S24. Steps S23 and S24 in the third embodiment are processed in the same manner as steps S23 and S24 in the first embodiment.

  In the third embodiment, the following effects can be obtained.

  In the third embodiment, as described above, when the smartphone 71 and the TV 2 are performing the mirroring operation on the SOC 71 of the remote controller 7, the smartphone 6 is based on the remote control operation information received by the touch panel 73. Is configured to perform operation control on the speaker 23 of the TV 2. As a result, the user confirms the mirroring image and the mirroring sound output to the display unit 22 of the TV 2 and controls the contents of the mirroring image and the mirroring sound by operating the remote control 7 in the same manner as a normal remote control operation. The remote controller can be operated using the remote controller, and the remote controller for performing the operation control on the speaker 23 of the TV 2 can be performed using the remote controller 7 in the same manner as the normal remote controller operation. Other effects of the operation system 102 according to the third embodiment are the same as those of the operation system 100 according to the first embodiment.

(Fourth embodiment)
Next, the configuration of the operation system 103 according to the fourth embodiment will be described with reference to FIG. 1, FIG. 16, and FIG. In the fourth embodiment, the hovering operation which is an operation in the air on the touch panel 95 of the touch panel 95 by the user is accepted and a hovering cursor (reference numeral B in FIG. 16) indicating the position coordinates corresponding to the three-dimensional position coordinate information is received. The image is superimposed on the mirroring image and displayed on the TV 2.

  As illustrated in FIG. 1, the operation system 103 according to the fourth embodiment includes a smartphone 9, and the smartphone 9 includes an SOC 91. The SOC 91 includes a CPU 92, a coordinate information generation unit 93, an image generation unit 94, and a touch panel 95.

  Here, in the fourth embodiment, as shown in FIGS. 1 and 16, the touch panel 95 includes a projected capacitive touch panel, and the touch panel 95 surface of the touch panel 95 by the user is detected by detecting the capacitance. It is configured to accept a hovering operation that is an upper airborne operation.

  As shown in FIGS. 1 and 16, when the touch panel 13 receives a hovering operation, the coordinate information generation unit 93 has three-dimensional position coordinate information (XYZ coordinates corresponding to the position of the user's finger or the like). Is configured to generate The image generation unit 94 is configured to superimpose a hovering cursor (reference numeral B in FIG. 16 and reference numeral C in FIG. 17) indicating the position coordinates corresponding to the three-dimensional position coordinate information on the mirroring image. Is configured to perform control to display the mirroring image and the hovering cursor on the TV 2.

  Then, as shown in FIG. 17, when the user touches the surface of the touch panel 95 (when the Z coordinate is 0), the CPU 92 is a hovering cursor (showing a position coordinate corresponding to the two-dimensional position coordinate information). 17 is configured to be superimposed on the mirroring image. Also, as shown in FIG. 17, when the user touches the touch panel 95 on the volume up button (A in FIG. 17) (when the Z coordinate is 0), the CPU 92 increases the volume of the TV 2. On the other hand, if the hovering operation (when the Z coordinate is other than 0) is performed in the space on the touch panel 95 on the volume up button (A in FIG. 17) on the volume up button (A in FIG. 17), the volume of the TV 2 is increased. It is configured to perform control. Other configurations of the operation system 103 according to the fourth embodiment are the same as those of the operation system 100 according to the first embodiment.

  In the fourth embodiment, the following effects can be obtained.

  In the fourth embodiment, as described above, the touch panel 95 receives a hovering operation that is an operation in the air on the touch panel 95 surface of the touch panel 95 by the user in the remote control operation input image displayed on the display unit 12. The SOC 91 is configured to control the hovering cursor (FIG. 16B and FIG. 17C) indicating the position coordinates corresponding to the hovering operation to be superimposed on the mirroring image and displayed on the TV 2. To do. Thereby, the user can visually recognize the hovering cursor indicating the position coordinates corresponding to the hovering operation displayed on the TV 2. As a result, the user can perform operation control on the TV 2 without visually recognizing the display unit 12 of the smartphone 9. Other effects of the operation system 103 according to the fourth embodiment are the same as those of the operation system 100 according to the first embodiment.

(Fifth embodiment)
Next, the configuration of the operation system 104 according to the fifth embodiment will be described with reference to FIG. 1, FIG. 18, and FIG. In the fifth embodiment, unlike the operation system 103 according to the fourth embodiment configured to receive a hovering operation that is an aerial operation on the touch panel surface of the touch panel by the user, based on the touch operation on the touch panel surface. It is configured to generate three-dimensional position coordinate information.

  As shown in FIG. 1, the operation system 104 by 5th Embodiment contains the smart phone 9a, and the smart phone 9a contains SOC91a. The SOC 91a includes a CPU 92a, a coordinate information generation unit 93a, an image generation unit 94, and the touch panel 13.

  As illustrated in FIG. 18, the touch panel 13 is configured to accept a touch operation on the touch panel 13 surface of the touch panel 13 by the user in the image displayed on the display unit 12, similarly to the operation system 100 according to the first embodiment. Has been.

  Here, in the fifth embodiment, as illustrated in FIG. 18, the coordinate information generation unit 93 a performs a hovering operation that is an aerial operation on the surface of the touch panel 13 based on the position information of the touch operation received by the touch panel 13. As shown, three-dimensional coordinate information (XY coordinate corresponding to the touch position and Z-axis coordinate corresponding to a preset non-zero value) is generated. Until the predetermined time elapses, the coordinate information generation unit 93a is configured to generate three-dimensional coordinate information. On the other hand, after the touch panel 13 is touched, the coordinate information generation unit 93a continues for the predetermined time. When the touch operation is performed, the touch operation is a two-dimensional touch operation on the surface of the touch panel 13 of the touch panel 13 (the XY coordinate corresponds to the touch position, and the Z-axis coordinate is 0). It is configured to generate position coordinate information. The touch operation at the same position for a predetermined time is an example of the “predetermined operation” in the present invention.

  As shown in FIGS. 18 and 19, as in the operation system 103 according to the fourth embodiment, the SOC 91a includes three-dimensional position coordinate information (position separated from the touch panel 13) and two-dimensional position coordinate information (touch panel). The hovering cursor (reference numeral B in FIG. 18 and reference numeral C in FIG. 19) indicating the position coordinates corresponding to the position 13 is superposed on the mirroring image and displayed on the TV 2. Other configurations of the operation system 104 according to the fifth embodiment are the same as those of the operation system 100 according to the first embodiment.

  In the fifth embodiment, the following effects can be obtained.

  In the fifth embodiment, as described above, when the SOC 91a is detected by the touch panel 13 as a touch operation, the touch operation is a three-dimensional position as a hovering operation that is an operation in the air on the touch panel 13 surface of the touch panel 13. In addition to generating coordinate information, when a predetermined operation is received after the touch operation is detected, two-dimensional position coordinate information is generated as a touch operation on the touch panel 13 of the touch panel 13. Control and a hovering cursor (FIG. 18 code B and FIG. 19 code C) indicating a position coordinate corresponding to at least the three-dimensional position coordinate information of the three-dimensional position coordinate information and the two-dimensional position coordinate information for mirroring It is configured to perform control to superimpose on the image and display on the TV 2. Thereby, even if it is a case where the touch panel 13 does not have the capability of detecting the user's finger in the air on the touch panel 13, the hovering operation can be accepted. As a result, since the user can visually recognize the hovering cursor indicating the position coordinates corresponding to the hovering operation displayed on the TV 2, the user can operate the TV 2 without viewing the display unit 12 of the smartphone 9a. Control can be performed. Other effects of the operation system 104 according to the fifth embodiment are the same as those of the operation system 100 according to the first embodiment.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description of the embodiments but by the scope of claims for patent, and further includes all modifications within the meaning and scope equivalent to the scope of claims for patent.

  For example, in the first to fifth embodiments, as an example of the information processing device, the present invention is applied to a smartphone. However, the present invention is not limited to this. The present invention is also applicable to information processing devices other than smartphones. For example, it can be applied to a tablet.

  In the first to fifth embodiments, an example in which the present invention is applied to a TV as an example of an image / audio output device has been described. However, the present invention is not limited to this. The present invention is also applicable to image / audio output devices other than TV. For example, a projector or a device combining a set top box, a display device, and an audio output device may be used.

  In the first embodiment and the third embodiment, when the smartphone and the TV are in a mirroring operation, an image displayed on the display unit of the smartphone is displayed as one of the mirroring image and the remote control operation input image. However, the present invention is not limited to this. In this invention, you may comprise so that the image displayed on the display part of a smart phone may display the image for mirroring, transform | convert the image for remote control operation input into semi-transparency, and to display it superimposed.

  In the third embodiment, an example in which the present invention is applied to a remote controller as an example of the first information processing device has been described. However, the present invention is not limited to this. The present invention may be configured to be applicable to the first information processing device other than the remote controller. For example, a smartphone separate from the smartphone that is performing the mirroring operation may be provided and applied as the first information processing device.

  In the first to fifth embodiments, the wireless LAN communication unit is used as the operated device side communication unit and the operation device side communication unit of the present invention. However, the present invention is not limited to this. For example, a configuration including a wired communication unit may be used.

  Further, in the first to fifth embodiments, the example in which the touch operation and the hovering operation are used as the user operation input method received by the operation reception unit of the present invention has been described. For example, it may be configured to further include a gyro sensor or the like to detect the movement of the smartphone body and accept the operation.

  In the first to fifth embodiments, an example in which a projected capacitive touch panel is included as the touch panel unit has been described, but the present invention is not limited thereto. In the present invention, a touch panel other than the projected capacitive touch panel may be used. For example, an electromagnetic induction touch panel or a resistive film touch panel may be used.

  Moreover, although the smart phone or the remote control showed the example which transmits the signal for performing audio | voice adjustment of TV as a remote control operation information signal in 1st-5th embodiment, this invention is not limited to this. In the present invention, when the remote control operation information signal is a signal for controlling the smartphone itself, the remote control operation information signal may not be transmitted to the TV 2.

  In the first to fifth embodiments, an example in which the smartphone or the remote controller performs the audio adjustment control of the TV as the TV operation control has been described, but the present invention is not limited thereto. In the present invention, control other than TV sound adjustment may be performed as TV operation control. For example, the image quality adjustment of the TV may be controlled.

  In the first to fifth embodiments, for convenience of explanation, the processing of the control unit of the present invention has been described using a flow-driven flowchart that performs processing in order along the processing flow, but the present invention is not limited thereto. Not limited. In the present invention, the processing operation of the control unit may be performed by event-driven (event-driven) processing that executes processing for each event. In this case, it may be performed by a complete event drive type or a combination of event drive and flow drive.

1, 5, 6, 9, 9a Smartphone (information processing equipment)
2, 8 TV (video / audio output equipment)
7 Remote control (first information processing equipment)
11, 51, 61, 91, 91a SOC (control unit)
12 Display unit 13, 95 Touch panel (operation receiving unit)
14, 65 Wireless LAN communication unit (communication unit)
26 Wireless LAN communication unit (Video / audio output device side communication unit)
22 Display (Image / Audio Output Equipment Display)
23 Speaker (Audio output unit on the audio output device side)
71 SOC (first information processing equipment side controller)
72 Wireless LAN communication section (first information processing equipment side communication section)
73 Touch panel (remote control accepting part)
100, 101, 102, 103, 104 Operation system

Claims (10)

An information processing device having a function of mirroring a mirroring image and a mirroring sound with respect to an external image / sound output device and a function as a remote control,
A communication unit that communicates with the external video and audio output device;
A display unit for displaying a remote control operation input image different from the mirroring image and the mirroring image;
An operation accepting unit that accepts an operation by a user on the remote control operation input image displayed on the display unit;
A control unit that performs the remote control operation control based on operation information of the remote control operation received by the operation receiving unit;
The control unit transmits the mirroring image and the mirroring sound via the communication unit, and performs the mirroring operation to display the mirroring image on the external image / sound output device, while the display unit Displays the image for remote control operation input, performs operation control on the information processing device itself based on the received operation information of the remote control operation, and controls operation on the external video / audio output device An information processing device configured to perform   In the non-mirroring operation, the control unit can display the remote control operation input image on the display unit as in the mirroring operation, and based on the received operation information of the remote control operation, the information processing device itself The information processing device according to claim 1, wherein the information processing device is configured to perform operation control on the external video and audio output device and to perform operation control on the external video and audio output device. The operation reception unit includes a touch panel,
The information processing apparatus according to claim 1, wherein the touch panel is configured to receive a touch operation by a user in the remote control operation input image displayed on the display unit.   When the touch panel detects the touch operation, the control unit generates the three-dimensional position coordinate information as the hovering operation that is an operation in the air on the touch panel surface of the touch panel, and When a predetermined operation is received after the touch operation is detected, the touch operation is generated as two-dimensional position coordinate information as the touch operation on the touch panel surface of the touch panel; A hovering cursor indicating a position coordinate corresponding to at least the three-dimensional position coordinate information of the three-dimensional position coordinate information and the two-dimensional position coordinate information is superimposed on the mirroring image, and the external image sound The information processing device according to claim 3, wherein the information processing device is configured to perform control to be displayed on the output device. The operation reception unit includes a touch panel,
The touch panel is configured to accept a hovering operation that is an operation in the air on the touch panel surface of the touch panel by a user in the remote control operation input image displayed on the display unit,
The control unit is configured to perform control to superimpose a hovering cursor indicating a position coordinate corresponding to the hovering operation on the mirroring image and to display the hovering cursor on the external image / audio output device. The information processing apparatus according to any one of 1 to 4.   The control unit transmits first mirroring image and the mirroring sound via the communication unit, and executes first mirror software for performing a mirroring operation for displaying the mirroring image on the external image / sound output device. While executing, the remote control operation input image is displayed on the display unit, and the information processing device itself is controlled based on the received operation information of the remote control operation, and the external video / audio output The information processing device according to claim 1, configured to execute second application software that performs operation control on the device.   The information processing apparatus according to claim 6, wherein the first application software and the second application software are configured by a single application software integrated with each other.   The control unit displays the image for remote control operation on the display unit without displaying the image for mirroring while displaying the image for mirroring on the external image / sound output device. The information processing device according to claim 6, wherein the information processing device is configured to receive the information. Image and audio output equipment,
An information processing device having a function of mirroring a mirroring image and a mirroring sound with respect to the image and sound output device and a function as a remote control;
The image / audio output device includes an image / audio output device side communication unit that communicates with the information processing device, an image / audio output device side display unit that displays the mirroring image, and an image / audio output device that outputs the mirroring sound. Side audio output unit,
The information processing device includes: an information processing device side communication unit that communicates with the video and audio output device; an information processing device side display unit that displays a remote control operation input image different from the mirroring image and the mirroring image; The information processing device side operation receiving unit that receives an operation by a user on the remote control operation input image displayed on the information processing device side display unit, and the remote control operation operation information received by the information processing device side operation receiving unit Based on the information processing equipment side control unit for performing the remote control operation control,
The information processing device side control unit transmits the mirroring image and the mirroring sound to the image and sound output device via the information processing device side communication unit and the image and sound output device side communication unit. While performing a mirroring operation for displaying the image for mirroring on the image / audio output device side display unit, the information processing device side display unit displays the image for remote control operation input on the received operation information of the remote control operation. An operation system configured to perform operation control on the information processing device itself and to perform operation control on the audio / video output device side audio output unit. Image and audio output equipment,
A first information processing device having a remote control function of the video and audio output device;
A second information processing device having a function of mirroring a mirroring image and a mirroring sound with respect to the image and sound output device;
The video / audio output device includes a video / audio output device-side communication unit that communicates with the first information processing device and the second information processing device, a video / audio output device-side display unit that displays the mirroring image, and the mirroring. Including an image / sound output device side sound output unit that outputs sound for use,
The first information processing device includes a first information processing device side communication unit that communicates with the video / audio output device and the second information processing device, a remote control operation reception unit that receives an operation by a user, and the remote control operation reception unit. A first information processing device-side control unit that performs the remote control operation control based on the received remote control operation information,
The second information processing device includes: a second information processing device side communication unit that communicates with the video / audio output device and the first information processing device; a second information processing device side display unit that displays the mirroring image; Via the second information processing device side communication unit and the image / sound output device side communication unit, the mirroring image and the mirroring sound are transmitted to the image / sound output device, and the image / sound output device side display A second information processing device side control unit configured to perform a mirroring operation for displaying the mirroring image on the unit,
When the second information processing device and the video / audio output device are performing the mirroring operation, the first information processing device side control unit is configured such that the remote control operation reception unit of the first information processing device Based on the received operation information of the remote control operation, the operation control is performed on the second information processing device, and the operation control is performed on the image / sound output device side sound output unit. Operation system.

Images