JP2016127425A - Operation ui providing method and program and operation target device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable an operation target device to allow an operation device to display a predetermined operation UI as necessary while displaying content in an environment where the operation target device independently controls the display of the content.SOLUTION: A communication part 61 of an operation target device (viewing device) SK which controls the display of content is configured to, when the operation target device SK provides an operation UI to an operation device SP, transmit operation UI configuration information corresponding to the display scene of the content from the operation target device SK to the operation device SP. A display control part 53 of the operation device SP is configured to allow a display device 26 of the own device to display the operation UI in accordance with the operation UI configuration information. The processing of the communication part 61 and the processing of the display control part 53 are repeated in accordance with the change of the display scene.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a method of providing an operation UI (operation user interface) from an operated device that controls display of content to an operation device. “Operated device” refers to a device that is operated in response to a signal from the operating device. “Operating device” refers to a device for operating an operated device.

  Conventionally, techniques for transferring and displaying content display screens between devices are known, and there are several types of standards.

  Miracast (registered trademark) established by the Wi-Fi Alliance is a technology for wirelessly rendering display output including content rendering results and application GUI (Graphical User Interface). The communication path is transmitted by Wi-Fi Direct (registered trademark), the protocol is transmitted by RTP / RTSP, and the output UI (User Interface) is compressed to MPEG2-TS (H.264 / LPCM / AAC / AC3).

  Apple's AirPlay (registered trademark) is a function for streaming music, videos, and images played on iTunes, iPhone, iPod touch, and iPad (all registered trademarks) on other devices.

  MirrorLink (registered trademark) of Car Connectivity Consortium is a standard for communication connection between mobile devices such as smartphones and in-vehicle AV devices.

  In addition, DLNA (registered trademark) guidelines and the like are known.

  These techniques are used in order for one device and the other device to display the same screen by transferring the screen displayed by one device to the other device for display. However, depending on the content of the screen displayed by one device, the other device may not display the same screen but may want to display a screen different from the one device.

  It is an object of the present invention to display a predetermined operation UI on an operating device as appropriate when the operated device is displaying content in an environment in which the operated device actively controls display of the content.

  The present application that solves this problem is characterized by a method of providing an operation UI from an operated device that controls display of content to an operating device. First, in the transmission step, the operation UI configuration information corresponding to the content display scene is transmitted from the operated device to the operating device. In the display step, the operation device displays the operation UI on the display device of the own device according to the operation UI configuration information. Then, the transmission step and the display step are repeated according to the change of the display scene.

  In the view mode and the hybrid mode, which will be described later, the “transmission step” corresponds to processing executed by the communication unit of the viewing device. The “display step” corresponds to a process executed by the display control unit of the controller device.

  According to the present invention, operation UI configuration information prepared in advance for each display scene is transmitted from the operated device to the operating device according to the display scene of the content displayed by the operated device. The operation device displays a predetermined operation UI on the display device of the own device according to the operation UI configuration information received from the operated device.

  For this reason, according to the present invention, in an environment where the operated device actively controls display of the content, the operated device can appropriately display the predetermined operation UI while the content is being displayed.

FIG. 1 is an overall configuration diagram of a content viewing system according to an embodiment of the present invention. FIG. 2 is a configuration diagram of a projection mode executed by the content viewing system of FIG. FIG. 3 is an explanatory diagram of the projection mode of FIG. FIG. 4 is a configuration diagram of image countermeasure processing executed by the projection mode of FIG. FIG. 5 is a flowchart of image countermeasure processing executed by the viewing device SK of FIG. FIG. 6 is a configuration diagram of a sensor emulation function executed by the content viewing system of FIG. 7 is a configuration diagram of a view mode and a hybrid mode executed by the content viewing system of FIG.

  Hereinafter, an embodiment of a content viewing system to which the present invention is applied will be described.

[System overall configuration]
FIG. 1 is an overall configuration diagram of a content viewing system. A viewing device SK is connected to the television monitor MON. The viewing device SK outputs a video signal and an audio signal to the monitor MON. The viewing device SK performs wireless communication conforming to the operation device SP and Wi-Fi (Wireless Fidelity) via an access point AP of a wireless LAN (Local Area Network). The access point AP is connected to a WAN (Wide Area Network) by wire. The WAN is provided with a content server CS, and the controller device SP communicates with the content server CS via the AP. The viewing device SK also communicates with the content server CS via the AP.

  Communication between the operation device SP and the viewing device SK is permitted after mutual authentication of the devices and is performed via a logical communication path. Similarly, the communication between the operation device SP and the content server CS and the communication between the viewing device SK and the content server CS are permitted after mutual authentication of the devices and are performed via the logical communication path.

  In the present embodiment, the controller device SP is obtained by installing a predetermined application program (application) on a smartphone having a Wi-Fi interface. In addition, the viewing device SK is housed in a stick-shaped housing having the same size as a commercially available USB memory. The housing has a built-in Wi-Fi interface and an HDMI (High-Definition Multimedia Interface, registered trademark) terminal for video / audio output.

  The user controls the operation of the viewing device SK by operating the operation device SP. At this time, in order to improve user convenience, a projection mode, a view mode, a hybrid mode, and a sensor emulation function are used.

  The projection mode is a mode in which the same screen as the screen displayed on the television monitor MON by the viewing device SK is synchronously displayed on the display device of the operating device SP. The view mode is a mode in which a different UI is displayed on the display device of the operation device SP according to the scene of the screen displayed on the television monitor MON by the viewing device SK. The hybrid mode is a mode in which the projection mode or the view mode is selectively executed according to the scene of the screen displayed on the television monitor MON by the viewing device SK. The sensor emulation function is a function that allows the application program of the viewing device SK to use the functions of the various sensors by reproducing the functions of the various sensors provided in the operation device SP on the viewing device SK. In the present embodiment, the functions of various sensors provided in the mobile terminal (smartphone) as the operation device SP are reproduced on the viewing device SK. Each of the above modes and functions will be described in detail later.

  The operating device SP includes a processing device that realizes various functions by executing a program, and a storage device that stores information used by the processing device for processing. The operation device SP includes an input device for a user to input information and a display device for displaying information to the user. Furthermore, the operating device SP includes a communication device for communicating with the viewing device SK and the content server CS. In this embodiment, a touch panel is provided as an input device and display device. The communication device is a Wi-Fi interface as described above.

  On the other hand, the hardware of the viewing device SK includes a processing device that realizes various functions by executing a program, and a storage device that stores information used by the processing device for processing. Also, an input interface (input I / F) for connecting an input device and a display interface (display I / F) for connecting a display device are provided. Furthermore, a communication device for communicating with the operation device SP and the content server CS is provided. In the present embodiment, the input I / F is a USB terminal and is provided mainly for the purpose of connecting a USB device during maintenance. As described above, the display I / F is an HDMI (registered trademark) terminal, and the communication device is a Wi-Fi interface.

  In the present embodiment, a common platform is mounted on the operation device SP and the viewing device SK. The platform is, for example, Android (registered trademark). In the following, each mode and function executed on this platform will be described in detail.

[Projection mode]
In the projection mode, the screen displayed on the television monitor MON by the viewing device SK and the screen displayed on the display device of the operation device SP are synchronized. That is, in this mode, the controller device SP synchronously displays the same screen as the screen displayed by the viewing device SK. A configuration diagram of the projection mode is shown in FIG. The storage device 11 of the viewing device SK stores an original drawing command, a drawing library, and a revised drawing command.

The “original drawing command” is a drawing command including an instruction to draw a plurality of figures having an overlapping order.
The “figure having the overlapping order” may be a plurality of figures drawn on different layers set with the overlapping order, or a plurality of figures set with the overlapping order for each figure.
The “drawing library” is a drawing library installed in the platform.
The “revised drawing command” is a drawing command for drawing the appearance of one screen drawn by the original drawing command with a primitive.
“Primitive” is a basic figure component such as a point, a line, a triangle, or a rectangle.
“Appearance of a single screen” refers to a single picture that can be visually recognized when a plurality of figures are drawn two-dimensionally in consideration of the overlapping order.

  When the viewing device SK draws a screen to be displayed on the television monitor MON, the original drawing command is passed to the command generation unit 12. The command generation unit 12 uses the drawing library to generate a revised drawing command based on the original drawing command and stores it in the storage device 11 (command generation step). For example, as shown in FIG. 3A, when the original drawing command is to draw a square Q filled in front of a long straight line L1, the dotted line portion of the straight line L1 on the back of the square is hidden and cannot be seen. The appearance of the drawn screen is as shown in FIG. Therefore, the revised drawing command is generated as a drawing command for drawing the quadrangle Q shown in FIG. 3B and two short straight lines L2 and L3 connected to both sides thereof by primitives. As a result, the total number of pixels rendered by the revised rendering command is reduced compared to the total number of pixels rendered by the original rendering command.

  When the command generation unit 12 stores a revised drawing command for drawing one screen in the storage device 11, the transmission unit 13 sends the revised drawing command for drawing one screen from the storage device 11. Read and transmit to the controller device SP (transmission step).

  Further, the command execution unit 14 reads out a redrawing command for drawing one screen from the storage device 11 and inputs it to the drawing driver of the display processing unit 15. The drawing driver of the display processing unit 15 draws one screen based on the revised drawing command and displays it on the television monitor MON (display step).

  In parallel with this, in the controller device SP, the receiving unit 23 receives the revised drawing command transmitted from the viewing device SK and stores it in the storage device 21. The command execution unit 24 of the controller device SP reads out a redrawing command for drawing one screen from the storage device 21 and inputs it to the drawing driver of the display processing unit 25. In this embodiment, the platform of the viewing device SK and the platform of the operation device SP are the same, and the specifications of the drawing drivers of both devices are also the same. The drawing driver of the display processing unit 25 draws one screen based on the revised drawing command and displays it on the display device 26 (display step).

  According to this projection mode, a redrawing command for drawing one screen with primitives is transferred from the viewing device SK to the operating device SP, and the viewing device SK and the operating device SP perform distributed drawing processing based on the same redrawing command. I do. For this reason, it is possible to improve the display synchronization speed between devices as compared with the conventional method of transferring drawing pixel information of a screen drawn by one device to the other device and displaying the same screen between the devices. it can.

  In addition, according to the present embodiment, since the platform of the viewing device SK and the platform of the operating device SP are the same, the application program of the operating device SP inputs the revised drawing command received from the viewing device SK as it is to the drawing driver. Thus, the same screen as that of the viewing device SK can be displayed. For this reason, the application software of the operating device SP can be developed without being bound by the application program of the viewing device SK. Similarly, the application program of the viewing device SK can display the same screen on the operating device SP by transmitting the revision drawing command for the viewing device SK as it is to the operating device SP. Therefore, the application program for the viewing device SK can be developed without being bound by the application program for the operation device SP.

[Image processing in projection mode]
When the viewing device SK displays an image, the image data must be transferred from the viewing device SK to the operation device SP in the projection mode. However, when image data is transferred from the viewing device SK to the operation device SP, the data capacity of the image is relatively large and it takes time to transfer, so the display synchronization speed between the devices is deteriorated. Therefore, image countermeasure processing that solves this problem is installed in the projection mode.

  FIG. 4 is a configuration diagram of a projection mode equipped with image countermeasure processing. Compared with the projection mode of FIG. 2, in FIG. 4, the storage device 11 of the viewing device SK and the storage device 21 of the operation device SP store the association between the image data and the image identifier, respectively. The “image identifier” is a unique identifier for specifying image data, and a value that does not overlap with other image data is designated.

  When the instruction generation unit 12 of the viewing device SK detects that the original drawing instruction includes drawing of an image in the projection mode, the instruction generation unit 12 executes the image countermeasure process shown in FIG. In S <b> 1, the command generation unit 12 determines whether an image identifier has already been assigned to the image data of the image to be drawn. That is, it is determined whether or not the association between the image data and the image identifier is already registered in the storage device 11.

  As a result, when the image identifier is not assigned to the image data, that is, when the image is drawn for the first time, a unique image identifier is assigned to the image data in S2, and the association between the image data and the image identifier is performed. Is registered in the storage device 11. In S3, the command generation unit 12 embeds the image identifier in the revised drawing command generated based on the original drawing command. That is, an image identifier corresponding to the image is embedded in the drawing command sentence so that the revised drawing command draws a predetermined image in a predetermined area on the screen.

  In S4, the transmission unit 13 associates the image data in S2 and the image identifier, and transmits them to the controller device SP. In addition, the transmission unit 13 transmits a revision drawing command in which the image identifier is embedded to the operation device SP. Here, the transmission of the image data associated with the image identifier may be performed simultaneously with the transmission of the redrawing command. Alternatively, the transmission of the image data associated with the image identifier can be performed between the processes of S2 and S3. The instruction execution unit 14 of the viewing device SK reads the image data stored in association with the image identifier from the storage device and passes it to the drawing driver of the display processing unit 15 when the image identifier is embedded in the revision drawing command. . The drawing driver of the display processing unit 15 renders one screen including the image based on the revised drawing command, and displays it on the television monitor MON.

  On the other hand, if a unique image identifier has already been assigned to the image data of the image to be rendered in S1, that is, if the image has been rendered for the second time or later, the command generation unit 12 determines in S5 A unique image identifier associated with the image data is read from the storage device 11. In S6, the command generation unit 12 embeds the image identifier in the revised drawing command generated based on the original drawing command. That is, an image identifier corresponding to the image is embedded in the drawing command sentence so that the revised drawing command draws a predetermined image in a predetermined area on the screen. In S <b> 7, the transmission unit 13 transmits a revision drawing command in which the image identifier is embedded to the controller device SP.

  On the other hand, in the controller device SP, the receiving unit 23 receives the image data and the image identifier transmitted from the viewing device SK, and stores them in the storage device 21 in association with each other. In this embodiment, the storage destination is a cache area. In addition, the receiving unit 23 receives the revision drawing command in which the image identification information is embedded from the operation device SP, and stores it in the storage device 21. When an image identifier is embedded in the revised rendering command, the command execution unit 24 of the operating device SP passes the image data in the storage device 21 stored in association with the image identifier to the rendering driver of the display processing unit 25. . The drawing driver of the display processing unit 25 renders one screen including the image based on the revised drawing command, and displays it on the display device of the operating device SP.

  According to the above image countermeasure processing, the image data transferred from the viewing device SK to the operation device SP is stored in the storage device 21 of the operation device SP. When the same image is drawn again, the image identifier is passed from the viewing device SK to the operation device SP, and the operation device SP reads out the image data corresponding to the image identifier from its own storage device 21 and uses it. For this reason, when rendering the same image again, there is no need to transmit image data from the viewing device SK to the operation device SP. Therefore, it is possible to minimize the frequency with which relatively large-capacity image data is transmitted between apparatuses. Thereby, in the projection mode, it is possible to suppress a decrease in display synchronization speed between apparatuses.

  In the projection mode, the screen display of the viewing device SK is projected onto the display device of the operating device SP. When the screen displayed on the television monitor MON by the viewing device SK is projected onto the display device of the operation device SP, if the real time property of the projection cannot be secured, the operator feels uncomfortable, and as a result, the operation device SP functions as a controller. Can't play a role. On the other hand, according to the projection mode of the present embodiment, the screen displayed by the viewing device SK can be displayed on the display device of the operation device SP while securing the real-time performance of projection between two physically separated devices. .

  Further, according to the projection mode, the UI provided by the application program on the viewing device SK side can be completely reproduced on the display device on the operation device SP. Thereby, an environment suitable for operating an application program that mainly operates on the side of the viewing device SK connected to the television monitor MON from the side of the operating device SP such as a smartphone is prepared.

  In the above description of the projection mode and the image countermeasure processing, the command generation unit 12, the transmission unit 13, the command execution unit 14, and the display processing unit 15 of the viewing device SK are realized by the processing device of the viewing device SK executing a program. To do. The receiving unit 23, the command execution unit 24, and the display processing unit 25 of the operating device SP are realized by the processing device of the operating device SP executing a program.

[Sensor emulation]
Next, the sensor emulation function will be described with reference to FIG. The operating device SP is a mobile terminal such as a smartphone, and includes hardware that is operated on hand and is assumed to be a mobile terminal. Therefore, the operation device SP includes various hardware sensors that are not mounted on the viewing device SK. On the other hand, the viewing device SK is a device that is relatively small, light, and relatively inexpensive such as a USB memory, and has limited hardware as compared with the operation device SP. Therefore, it is considered to use the function of the hardware sensor of the operating device SP as a function of the viewing device SK (sensor emulation function). That is, the output of the hardware sensor of the operating device SP is transmitted to the viewing device SK in real time as if it were the output of the hardware sensor mounted on the viewing device SK. As a result, the application program of the viewing device SK can execute processing that uses the output of the hardware sensor in the same manner as the application program of the controller device SP.

  In FIG. 6, the controller device SP includes a transmitter 32 that transmits sensor value information based on the output of the hardware sensor 31 to the viewing device SK. On the other hand, the viewing device SK includes a receiving unit 42 that receives sensor value information from the operating device SP. In addition, the viewing device SK includes an input unit 43 that inputs input information based on the sensor value information to the virtual sensor device 41 corresponding to the hardware sensor 31. Furthermore, the viewing device SK includes an operation control unit 44 that controls the operation of the own device based on the output of the virtual sensor device 41.

  More specifically, in this embodiment, the sensor value information includes an association between a sensor identifier and a sensor value that distinguish the plurality of hardware sensors 31. The input unit 43 inputs input information to a virtual sensor device corresponding to the sensor identifier among the plurality of virtual sensor devices 41.

  In the present embodiment, the hardware sensor 31 of the operating device SP includes a pressure sensor, a proximity sensor, a temperature sensor, a humidity sensor, a GPS, a gyro sensor, an acceleration sensor, an illumination sensor, a magnetic field sensor, and the like. Each hardware sensor inputs the output signal value of the sensor to the corresponding sensor driver 35. Each sensor driver 35 converts the output signal value acquired from each sensor into a sensor value for an application program and passes it to the transmission unit 32. The transmission unit 32 generates sensor value information in which the acquired sensor value of each sensor is associated with the sensor identifier of each sensor by the processing of the application program, and transmits the sensor value information to the viewing device SK in real time. Here, the “sensor identifier” is a unique identifier for specifying one sensor device, and a value that does not overlap with another sensor device is designated.

  The receiving unit 42 of the viewing device SK receives the sensor value information transmitted from the controller device SP and passes it to the input unit 43. The viewing device SK includes a virtual sensor device 41 that operates when the processing device executes a program. The virtual sensor device 41 includes a plurality of virtual sensor devices corresponding to the hardware sensor 31 of the operating device SP. That is, in this embodiment, the virtual sensor device 41 includes virtual models such as a pressure sensor, a proximity sensor, a temperature sensor, a humidity sensor, a GPS, a gyro sensor, an acceleration sensor, an illumination sensor, and a magnetic field sensor. The virtual sensor device 41 is configured so that the OS of the viewing device SK can be recognized. Each virtual sensor device is assigned a sensor identifier. When receiving the sensor value information from the receiving unit 42, the input unit 43 inputs the sensor value to the virtual sensor device 41 corresponding to the sensor identifier. At this time, the input unit 43 converts the sensor value data so as to conform to the specification of the input terminal of the virtual sensor device 41 and inputs the sensor value data to the virtual sensor device 41.

  The virtual sensor device 41 is modeled to output the same output signal value as that of the hardware sensor 31 of the controller device SP that outputs the sensor value when the converted sensor value is input. For example, the virtual sensor device 41 of the viewing device SK corresponding to the pressure sensor 31 of the operating device SP outputs the same output signal value as the output signal value of the pressure sensor 31 in real time. Here, in the present embodiment, the platform of the operating device SP and the platform of the viewing device SK are the same. The sensor driver 35 of the operating device SP and the sensor driver 45 of the viewing device SP have the same configuration. The OS of the viewing device SK recognizes that the signal output from the virtual sensor device 41 is a signal generated from a hardware sensor mounted on the viewing device SK, and executes normal processing of the OS. The sensor value passed from the sensor driver 35 to the application program in the operating device SP is the same as the sensor value passed from the sensor driver 45 to the application program in the viewing device SK. The operation control unit 44 of the viewing device SK operates when the processing device executes the application program, and executes processing according to the sensor value acquired from the sensor driver 45.

  According to the sensor emulation function described above, the signal value output from the hardware sensor of the controller device SP is handled as if it were a signal value generated in real time in the viewing device SK, and the operation of the viewing device SK is controlled. Can do. Therefore, the application program of the viewing device SK can use the function of the hardware sensor that is not installed in the own device.

  In the above-described projection mode, the display screen of the viewing device SK is projected in real time on the touch panel of the operating device SP. In this state, the sensor emulation function is used to reflect the values of the touch sensor, pressure sensor, and the like on the viewing device SK in real time from the touch panel of the controller device SP. Accordingly, the UI projected from the viewing device SK onto the display device of the operating device SP can be operated from the operating device SP. Therefore, the combination of the projection mode and the sensor emulation function realizes a system that remotely operates the UI provided mainly by the viewing device SK from the operation device SP. This realizes separation of operation and UI. Further, the developer can create an application program for the viewing device SK on the assumption that a hardware sensor is mounted on the viewing device SK. In addition, the developer does not need to be aware of using the output of the hardware sensor of the operating device SP when creating the application program for the viewing device SK.

  In the above description of the sensor emulation function, the transmission unit 32 and the sensor driver 35 of the operating device SP are realized by the processing device of the operating device SP executing a program. In addition, the virtual sensor device 41, the receiving unit 42, the input unit 43, the operation control unit 44, and the sensor driver 45 of the viewing device SK are realized by the processing device of the viewing device SK executing a program. Here, the use of the sensor emulation function is not limited to the projection mode. Moreover, the kind of sensor is not ask | required.

[View mode]
Next, the view mode will be described. The projection mode described above is a mode in which the screen displayed by the viewing device SK is displayed synchronously on the display device of the operating device SP. However, depending on the scene displayed by the viewing device SK, it may be desired to display a UI different from the screen of the viewing device SK on the operation device SP. The view mode is a mode in which a UI different from the screen displayed by the viewing device SK is displayed on the operating device SP according to the scene of the screen displayed by the viewing device SK.

  FIG. 7 includes a view mode configuration. In FIG. 7, the viewing device SK provides an operation UI to the operation device SP. That is, the communication unit 61 of the viewing device SK transmits the operation UI configuration information corresponding to the content display scene from the viewing device SK to the operation device SP. The display control unit 53 of the operation device SP displays the operation UI on the display device 26 of the own device according to the operation UI configuration information. The communication unit 61 of the viewing device SK and the display control unit 53 of the operating device SP repeat the above operation every time the scene displayed by the viewing device SK is changed.

  The viewing device SK stores the association between the identifier of the scene to be displayed on the television monitor MON (display scene identifier) and the operation UI configuration information in the storage unit 62 for each scene. When the display scene is changed, the communication unit 61 of the viewing device SK reads the operation UI configuration information associated with the identifier of the display scene after the change from the storage unit 62 and transmits the operation UI configuration information to the operation device SP. The “display scene identifier” is an identifier unique to each content display scene.

  More specifically, in the viewing device SK, the display control unit 63 displays the display content acquired from the storage unit on the television monitor MON. During execution of the view mode, the scene change detection unit of the display control unit 63 detects a change in the display scene displayed by the display control unit 63. There are several possible methods for detecting changes in the display scene. For example, a method for detecting based on operation information for changing the display scene acquired from the controller device SP, a method for detecting that the display scene identifier of the currently displayed scene is different from the display scene identifier of the next scene to be displayed, Etc. are considered.

  When the display content is a UI, the display scene identifier may be assigned to each selectable object included in the UI. In the UI display, a scene in which one object is selected and a scene in which another object is selected are separate display scenes. In this case, different display scene identifiers may be assigned to the one object and the other objects.

  In the present embodiment, when the change of the display scene is detected, the scene change detection unit transmits the display scene identifier of the next scene to be displayed to the controller device SP via the communication unit 61. Alternatively, when the change of the display scene is detected, the scene change detection unit acquires the address (URL or the like) of the operation UI configuration information associated with the display scene identifier of the next scene to be displayed from the storage unit 62, and the communication unit 61 To the controller device SP. The operation device SP receives the display scene identifier or the address of the operation UI configuration information transmitted from the viewing device SK via the communication unit 51 and passes it to the mode selection unit 52. When the display scene identifier is acquired, the mode selection unit 52 adds the display scene identifier as a query character string to the address on the HTTP server of the viewing device SK and passes it to the HTTP client of the communication unit 51. Alternatively, when the address of the operation UI configuration information is acquired, the mode selection unit 52 passes the address on the HTTP server of the viewing device SK to the HTTP client of the communication unit 51.

  The HTTP client of the operation device SP directly transmits the operation UI configuration information associated with the display scene identifier or the entity of the operation UI configuration information to the address on the HTTP server of the viewing device SK passed from the mode selection unit 52. ,Request. The HTTP server of the viewing device SK receives this request, reads the operation UI configuration information corresponding to the request from the storage unit 62, and transmits it to the HTTP client of the operation device SP. In this embodiment, the operation UI configuration information is a program in a procedural language, and may be described in, for example, HTML5 or JavaScript (registered trademark), and may be accompanied by a style sheet such as CSS (Cascading Style Sheets).

  The HTTP client of the operation device SP receives the operation UI configuration information transmitted from the viewing device SK, and displays a UI corresponding to the display scene of the viewing device SK on the display device 26 according to the operation UI configuration information. The viewing device SK and the operation device SP immediately repeat the above process every time a change in the scene displayed by the viewing device SK is detected.

  In the present embodiment, when the user operates the UI displayed on the operation device SP based on the operation UI configuration information using the input device 54, the communication between the operation device SP and the viewing device SK that is necessary thereafter is performed. This is performed between the WebSocket client of the operating device SP and the WebSocket server of the viewing device SK.

  According to the above view mode, the UI displayed by the controller device SP can be dynamically changed according to the scene displayed by the viewing device SK. That is, the screen displayed by the viewing device SK and the screen of the operating device that controls the screen can be made different according to the scene. In addition, the user of the operation device SP can control the operation of the viewing device SK using a UI suitable for each scene. Therefore, variations of the screen that can be expressed by the entire system including the viewing device SK and the operation device SP are widened. Further, the software developer of the viewing device SK calls a simple SDK (software development kit) and registers the operation UI configuration information corresponding to the scene, thereby causing the operation device SP to display a desired UI corresponding to the scene. It is possible. That is, a programmable execution environment between two devices can be realized.

[Hybrid mode]
Next, a hybrid mode in which the above-described projection mode and view mode are selectively executed according to the display scene will be described. FIG. 7 is a configuration diagram of the hybrid mode. The storage unit 62 of the viewing device SK stores operation UI configuration information or projection mode instruction information in association with each display scene identifier. The “display scene identifier” and “operation UI configuration information” are as described above. The “projection mode instruction information” is information that instructs the operating device SP to execute the projection mode described above.

  The display control unit 63 of the viewing device SK displays the display content acquired from the storage unit on the television monitor MON. The display control unit 63 includes a scene change detection unit and a projection mode processing unit. The projection mode processing unit of the viewing device SK executes the above-described processing on the viewing device SK side in the projection mode. When the hybrid mode is executed, the scene change detection unit detects a change in the scene displayed by the display control unit 63. The method for detecting the change of the display scene is as described above.

  In the present embodiment, when the scene change detection unit detects a change in the display scene, the scene change detection unit refers to the storage unit 62, and whether the information associated with the display scene identifier of the next scene to be displayed is the operation UI configuration information or It is determined whether it is a projection mode instruction.

  As a result, when the information associated with the display scene identifier is the operation UI configuration information, the scene change detection unit transmits the display scene identifier to the controller device SP via the communication unit 61. Alternatively, the scene change detection unit acquires the address (URL or the like) of the operation UI configuration information associated with the display scene identifier from the storage unit 62 and transmits it to the operation device SP via the communication unit 61. On the other hand, if the information associated with the display scene identifier is the projection mode instruction information as a result of the determination, the scene change detection unit sends the projection mode viewing device described above to the projection mode processing unit of the viewing device SK. Instructs execution of processing on the SK side. In this case, the scene change detection unit transmits a projection mode execution instruction to the controller device SP via the communication unit 61.

  The operation device SP receives the display scene identifier, the address of the operation UI configuration information, or the projection mode execution instruction transmitted from the viewing device SK via the communication unit 51, and passes it to the mode selection unit 52. When the display scene identifier or the address of the operation UI configuration information is acquired, the mode selection unit 52 selects the view mode. On the other hand, the mode selection unit 52 selects the projection mode when acquiring the projection mode execution instruction.

  When the mode selection unit 52 selects the view mode, the process of the view mode described above is executed by the cooperation of the viewing device SK and the operation device SP. On the other hand, when the mode selection unit 52 selects the projection mode, the projection mode processing described above is executed by the cooperation of the viewing device SK and the operation device SP. The above processing is repeated every time the scene displayed on the viewing device SK is changed.

  According to the hybrid mode described above, the projection screen in the projection mode or the UI for the operation device in the view mode can be dynamically switched and displayed on the display device of the operation device SP according to the scene displayed by the viewing device SK. it can. Thereby, the controller device SP becomes a more advanced controller.

  In the above description of the view mode and the hybrid mode, the communication unit 61 and the display control unit 63 of the viewing device SK are realized by the processing device of the viewing device SK executing a program. In addition, the communication unit 51, the mode selection unit 52, and the display control unit 53 of the operating device SP are realized by the processing device of the operating device SP executing a program.

  Although the above is an embodiment of the present invention, the present invention is not limited to the above embodiment. In the above-described embodiment, the projection mode and the sensor emulation function are executed by the viewing device SK and the operation device SP, but are not limited thereto, and can be executed by one device and the other device. In the above embodiment, the view mode and the hybrid mode are also executed by the viewing device SK and the operation device SP. However, the view mode and the hybrid mode are not limited to this, and can be executed by the operation device and the operated device. The scope of the present invention is the scope of the invention described in the claims, and is not limited to the above embodiment.

DESCRIPTION OF SYMBOLS 11,62 Storage device of viewing device 12 Command generation unit of viewing device 13 Transmission unit of viewing device 14 Command execution unit of viewing device 15 Display processing unit of viewing device 21 Storage device of operation device 23 Reception unit of operation device Command execution unit 25 display processing unit 26 of operation device display device 31 of operation device hardware sensor 32 of operation device transmission unit 35 of operation device sensor driver 41 of operation device virtual sensor device 42 of viewing device reception unit 43 of viewing device Viewing device input unit 44 Viewing device operation control unit 45 Viewing device sensor driver 51 Operating device communication unit 52 Operating device mode selection unit 53 Operating device display control unit 54 Operating device input device 61 Viewing device communication unit 63 Display control unit AP of viewing device Access point CS Content server L1, L2, L3 Straight line MON TV monitor TA Q Rectangle SK Viewing device SP Operation device

Claims (6)

A method of providing an operation UI from an operated device that controls display of content to an operating device,
A transmission step of transmitting operation UI configuration information corresponding to the display scene of the content from the operated device to the operating device;
A display step in which the operation device displays an operation UI on a display device of the device according to the operation UI configuration information;
An operation UI providing method in which the transmission step and the display step are repeated according to the change of the display scene. The operation UI providing method according to claim 1,
In the transmission step, the operation UI configuration information or drawing command information for drawing the same screen as the display scene by a primitive is selectively transmitted from the operated device to the operating device according to the display scene,
In the display step, when the operation device receives the operation UI configuration information, the operation device displays the operation UI on the display device of the device according to the operation UI configuration information, and receives the drawing command information. A method of providing an operation UI, wherein the same screen as the display scene drawn by the operated device is displayed on the display device of the own device according to the information. An operated device that provides an operation UI to the operation device,
A storage unit that stores, for each display scene, an association between a display scene that the operated device displays on the display device and the operation UI configuration information;
A transmission unit that reads operation UI configuration information associated with the changed display scene from the storage unit and transmits the operation UI configuration information to the operation device when the display scene is changed;
An operated device that causes the operation device to display an operation UI based on the operation UI configuration information. In the operated device according to claim 3,
The storage unit stores, for each display scene, the operation UI configuration information or drawing command information for drawing the same screen as the display scene in a primitive manner,
When the display scene is changed, the transmission unit reads operation UI configuration information or drawing command information associated with the changed display scene from the storage unit, and transmits the operation UI configuration information or drawing command information to the operation device.
An operated device that causes the operation device to display an operation UI based on the operation UI configuration information or a screen based on the drawing command information. A program for an operated device that provides an operation UI to the operation device,
The operated device stores the association between the display scene to be displayed on the display device and the operation UI configuration information in the storage unit for each display scene,
When the display scene is changed, the operated device causes the processor of the operated device to execute a transmission step of reading the operation UI configuration information associated with the changed display scene from the storage unit and transmitting the information to the operating device. Operation UI providing program. In the operation UI providing program according to claim 5,
The storage unit stores, for each display scene, the operation UI configuration information or drawing command information for drawing the same screen as the display scene in a primitive manner,
In the transmission step, when the display scene is changed, the operation UI configuration information or the drawing command information associated with the changed display scene is read from the storage unit and transmitted to the operation device. Operation UI providing program for the device.

Images