TWI796734B - Display control method and display control system - Google Patents

Abstract

A display control method and a display control system are disclosed. The method includes: executing a remote control program of a mobile device; detecting a type of application program executed by a display device wirelessly connected to the mobile device; presenting a first type operation interface or a second type operation interface on an operation interface of the mobile device by the remote control program according to the type of the application program, such that an operation function provided by the operation interface satisfies an operation requirement of an application interface presented by the display device; detecting an operation behavior of the user on the operation interface by the remote control program; and sending a control signal by the remote control program to the display device according to the operation behavior, such that the display device operates the application interface according to the control signal.

Description

Display control method and display control system

The present invention relates to a display control technology, and in particular to a display control method and a display control system.

With the popularity of mobile devices such as smart phones and tablet computers, many application programs have developed versions suitable for different platforms for use by different types of electronic devices. Taking the Netflix audio-visual streaming platform as an example, users of mobile devices can download the mobile version of the Netflix application to the mobile device for use, while users of smart TVs or smart projectors can download the TV version of the Netflix application It can be used in smart TV or smart projector. After downloading and installing, mobile device users can operate the mobile version of the Netflix application installed in the mobile device through touch operations, while users of smart TVs or smart projectors can use the physical remote control to operate The Netflix app for TV.

On the other hand, as the use of smart TVs or smart projectors becomes more and more flexible, some communication platforms also allow users of smart TVs or smart projectors to communicate freely or download unofficial applications. However, once the user downloads the mobile version of the application to the smart TV or the smart projector, the user cannot simply use the physical remote control to control the smart TV or the smart projector to run the application normally.

In view of this, the present invention provides a display control method and a display control system, which can improve the above problems and improve more convenience in operation.

An embodiment of the invention provides a display control method for a mobile device. The mobile device is wirelessly connected to the display device. The display device runs the application program and presents the API of the application program on the display interface. The display control method includes: starting a remote control program; using the remote control program to detect the type of the application program running on the display device; according to the type of the application program, the remote control program The program presents one of the first type of operation interface and the second type of operation interface on the operation interface of the mobile device, so that the operation functions provided by the operation interface meet the operation requirements of the application programming interface; The remote control program detects the operation behavior of the user on the operation interface; and the remote control program transmits a control signal to the display device according to the operation behavior, so that the display device Operate the API.

An embodiment of the present invention further provides a display control system, which includes a display device and a mobile device. The display device is used for running the application program and presenting the API of the application program on the display interface. The mobile device is wirelessly connected to the display device. The mobile device is used to: start a remote control program; detect the type of the application program running on the display device by the remote control program; according to the type of the application program, control the The program presents one of the first type of operation interface and the second type of operation interface on the operation interface of the mobile device, so that the operation functions provided by the operation interface meet the operation requirements of the application programming interface; The remote control program detects the operation behavior of the user on the operation interface; and the remote control program transmits a control signal to the display device according to the operation behavior, so that the display device Operate the API.

Based on the above, according to the type of the application program currently running on the display device, the mobile device can present a suitable operation interface for the user to control the application program interface presented by the display device. In this way, the compatibility and convenience of using the mobile device to operate and control different types of application programs displayed on the remote display device can be improved.

FIG. 1 is a schematic diagram of a display control system according to an embodiment of the present invention. Please refer to FIG. 1 , the display control system 10 includes a display device 11 and a mobile device 12 . The display device 11 may include various electronic devices with functions of wireless communication, data processing and display, such as a smart TV, a smart display, or a smart projector.

In one embodiment, the display device 11 is a smart TV or a smart display. Therefore, the display device 11 may include at least one display. The display can be used to present image frames for users to watch through a display interface. For example, the display may include plasma (plasma) display, liquid crystal display (liquid crystal display, LCD), thin film transistor (Thin Film Transistor, TFT) liquid crystal display, light emitting diode (light emitting diode, LED) display or organic Light-emitting diode (Organic LED, OLED) displays or various types of displays.

In one embodiment, the display device 11 is a smart projector. Therefore, the display device 11 may include at least one projection interface. The projection interface can be used to project light onto the surface of a specific object (such as a projection screen), so as to present an image on the surface of the object for users to watch. For example, the projection interface may include at least one light emitting element for projecting the light.

The mobile device 12 can be a remote control device wirelessly paired with the display device 11 . For example, the mobile device 12 may include various portable electronic devices with wireless communication, data processing and display functions, such as smart phones, tablet computers, notebook computers, remote controllers or game consoles. The mobile device 12 can transmit wireless control signals to the display device 11 through wireless communication protocols such as Bluetooth or Wifi, so as to remotely control the system settings and/or display content of the display device 11 .

FIG. 2 is a functional block diagram of a mobile device according to an embodiment of the present invention. Referring to FIG. 1 and FIG. 2 , the mobile device 12 may include a processor 21 , a storage circuit 22 and an input/output (I/O) interface 23 . The processor 21 may be responsible for the whole or part of the operations of the mobile device 12 . For example, the processor 21 may include a central processing unit (CPU) or other programmable general-purpose or special-purpose microprocessors, digital signal processors (Digital Signal Processor, DSP), programmable controllers, special application Integrated circuit (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) or other similar devices or a combination of these devices.

The storage circuit 22 is coupled to the processor 21 and used for storing data. For example, the storage circuit 22 may include a volatile storage circuit and a non-volatile storage circuit. The volatile storage circuit is used for volatile storage of data. For example, the volatile storage circuit may include random access memory (Random Access Memory, RAM) or similar volatile storage media. The non-volatile storage circuit is used for non-volatile storage of data. For example, the non-volatile storage circuit may include a read-only memory (Read Only Memory, ROM), a solid state disk (solid state disk, SSD) and/or a traditional hard disk (Hard disk drive, HDD) or similar non-volatile Storage media.

The input/output interface 23 is coupled to the processor 21 and used for inputting and outputting signals. For example, the input/output interface 23 may include various input/output devices such as a touch screen, a touch panel, a network interface card, a speaker, a microphone, and/or a power supply circuit. The present invention does not limit the type of the input/output device.

In one embodiment, the mobile device 12 can communicate wirelessly with the display device 11 through the network interface card in the input/output interface 23 . For example, the mobile device 12 can send wireless signals to the display device 11 or receive wireless signals from the display device 11 through the network interface card in the input/output interface 23 .

In one embodiment, the display device 11 may be installed with a specific application program (also referred to as a first application program). The display device 11 can run the first application program and present an API corresponding to the first application program. Taking the Netflix video streaming platform as an example, the display device 11 can install and run a TV version or a mobile version of the Netflix application. In response to the installed and running application being a TV version of the Netflix application, the display device 11 may present an API corresponding to the TV version of the Netflix application on the display interface. Alternatively, in response to the installed and running application being a mobile version of the Netflix application, the display device 11 may present an API corresponding to the mobile version of the Netflix application on the display interface. For example, the API can be presented on the display of the display device 11 or projected on a projection screen.

In one embodiment, the remote control program 201 is stored in the storage circuit 22 . The processor 21 can run the remote control program 201 . In response to the type of the application program currently running on the display device 11 (i.e. the first application program), the remote control program 201 can provide a suitable operation interface and present this operation interface on the operation interface of the mobile device 12 (for example, present on the mobile device 12's touch screen). Afterwards, the remote control program 201 can receive user operations through the operation interface and send control signals to the display device 11 according to the user operations, so as to control the operation of the API presented by the display device 11 .

In one embodiment, the processor 21 can activate the remote control program 201 . After the remote control program 201 is activated, the remote control program 201 can actively detect the type of the application program (ie, the first application program) running on the display device 11 . According to the type of the first application program, the remote control program 201 can present a specific operation interface on the operation interface of the mobile device 12 (for example, present on the touch screen of the mobile device 12 ).

In one embodiment, the remote control program 201 can receive the identification information of the first application program from the display device 11 through the input/output interface 23 . For example, the identification information may include the program name, version information and/or code of the first application, which can be used to identify the first application. The remote control program 201 can detect the type of the first application program according to the identification information.

In one embodiment, the remote control program 201 can send a request to the display device 11 through the input/output interface 23 to request the identification information of the first application program. The display device 11 can receive this request. In response to the request, the display device 11 may provide the mobile device 12 with the identification information of the first application program.

In one embodiment, according to the type of the first application program, the remote control program 201 can present one of the first type of operation interface and the second type of operation interface on the operation interface of the mobile device 12, so that the operation interface The provided operating functions meet the operating requirements of the API presented by the display device 11 . Then, the remote control program 201 can detect the user's operation behavior on the operation interface and send a control signal to the display device 11 according to the operation behavior. The control signal can instruct or drive the display device 11 to operate the API currently displayed on the display device 11 according to the control signal.

In one embodiment, in response to the type of the first application being the first-type application, the remote control program 201 may present the first-type operation interface on the operation interface of the mobile device 12 . Alternatively, in response to the type of the first application being the second-type application, the remote control program 201 may present the second-type operation interface on the operation interface of the mobile device 12 .

In one embodiment, the first type of application is a mobile version of the application, and the second type of application is a TV version of the application. Generally speaking, the mobile version of the application program is suitable for mobile devices with touch screens such as smart phones or tablet computers, so that users can operate the application program through the touch operation function provided by the smart phone or tablet computer. Conversely, the TV version of the application is suitable for mobile devices without a touch screen, such as smart TVs or smart projectors, for users to operate the application through the button operation function provided by the physical remote control.

In one embodiment, assuming that the API currently presented by the display device 11 is the API corresponding to the mobile version of the Netflix application program (that is, the first type of application program), the remote control program 201 can use the first type The operation interface is presented on the operation interface of the mobile device 12 . At this time, the operation function provided by the operation interface (ie, the first type of operation interface) presented by the mobile device 12 can meet the operation requirements of the corresponding API of the mobile version of the Netflix application program (ie, the first type of application program) ( such as touch operation functions).

In one embodiment, assuming that the API currently presented by the display device 11 is the API corresponding to the TV version of the Netflix application program (that is, the second type of application program), the remote control program 201 can use the second type The operation interface is presented on the operation interface of the mobile device 12 . At this time, the operation functions provided by the operation interface (i.e., the second type of operation interface) presented by the mobile device 12 can meet the operation requirements of the corresponding API of the Netflix application program (i.e., the second type of application program) of the TV version ( e.g. traditional push-button functions).

In one embodiment, the remote control program 201 can adjust at least one operable object in the first-type operation interface presented by the mobile device 12 in response to a specific or different type of first-type application program running on the display device 11 Configuration. For example, according to the switching of the first type of application program currently running on the display device 11 from the mobile version of the Netflix application program to the mobile version of the web browser program (such as Google Chrome web browser), the remote control program 201 can change the first type of application program presented in the first type. Configuration of at least one operable object in the operation interface. Wherein, changing or adjusting the configuration of at least one operable object presented in the first type of operation interface may include adding an operable object presented in the first type of operation interface, deleting part of the operable objects presented in the first type of operation interface objects and/or change the functions of some operable objects presented in the first type of operation interface, etc.

In one embodiment, in response to the type of the application currently running on the display device 11 (i.e., the first application) being a first-type application, after detecting the user's operation behavior on the first-type operation interface, The remote control program 201 can transmit a control signal (also referred to as a first control signal) to the display device 11 through the input/output interface 23 . The first control signal can instruct or drive the display device 11 to operate the application program interface presented by the display device 11 according to a touch operation behavior. For example, the touch operation behavior can simulate the operation behavior performed by the user on the first type of operation interface. For example, the touch operation behavior may include at least one of screen sliding, screen zooming, and cursor movement for the API, but is not limited thereto.

In one embodiment, in response to the type of the application currently running on the display device 11 (i.e. the first application) being the second type of application, after detecting the user's operation behavior on the second type of operation interface, The remote control program 201 can transmit a control signal (also called a second control signal) to the display device 11 through the input/output interface 23 . The second control signal can instruct or drive the display device 11 to operate the API presented by the display device 11 according to a button operation action. For example, the button operation behavior reflects the operation behavior performed by the user on the second type of operation interface. For example, the button operation behavior may include at least one of object switching and object selection for the API, but is not limited thereto.

FIG. 3 is a schematic diagram of a first type of operation interface according to an embodiment of the present invention. Please refer to FIG. 3 , in an embodiment, the first type of operation interface includes an operation interface 31 . The operation interface 31 can be, for example, an operation interface on a mobile device provided for an application program that supports playing multimedia images (such as a mobile version of the Netflix application program). For example, a plurality of operable objects 310 , 320 , 330 and 341 - 344 can be presented in the operation interface 31 .

The operable object 310 includes a program type detection object, which is used to trigger the detection of the type of the first application program. For example, in one embodiment, after the remote control program 201 in FIG. 2 is activated, the remote control program 201 can instruct the mobile device 12 to present an initial operation interface on the touch screen of the mobile device 12 . After detecting that the user clicks on the operable object 310, the remote control program 201 can start to detect the type of the application program (i.e. the first application program) running on the display device 11 of FIG. 1 and present the operation according to the detection result interface 31 (or the operation interface 41 in FIG. 4 , or the operation interface 51 in FIG. 5 ).

The operable objects 320 include a plurality of virtual sliding objects. When it is detected that the user clicks or touches these virtual sliding objects, a corresponding control signal can be sent. In particular, these control signals can instruct the display device 11 to perform a screen sliding operation (such as sliding the screen up, down, left or right) on the presented API of the first type of application. The screen sliding operation is similar to the touch operation behavior caused by the user of the mobile device 12 operating the sliding gesture on the touch screen or touch panel.

It should be noted that, in one embodiment, after the display device 11 starts to play the multimedia image according to the operation performed by the user on the API of the first type of application, the function of the operable object 320 can be changed. For example, after starting to play the multimedia video, the operable object 320 can be changed to provide a playback control function of the multimedia video. For example, after starting to play a multimedia image, when it is detected that the user clicks on the operable object 320, the corresponding control signal can be sent to instruct the display device 11 to perform fast forward, rewind, pause/play, previous Play control functions such as the first and the next one.

The operable object 330 includes a volume adjustment object. When it is detected that the user clicks or drags the volume adjustment object, a corresponding control signal can be sent. In particular, these control signals can instruct the display device 11 to adjust the volume of the output audio signal.

The operable objects 341-344 include a plurality of system objects. For example, when it is detected that the user clicks on the operable object 341, the corresponding control signal sent may instruct the display device 11 to open the control panel or the control menu related to the first application program. For example, when it is detected that the user clicks on the operable object 342, a corresponding control signal can be sent to instruct the display device 11 to return to the main screen of the programming interface. For example, when it is detected that the user clicks on the operable object 343 , a corresponding control signal can be sent to instruct the display device 11 to return to the previous page. For example, when it is detected that the user clicks on the operable object 344 , the corresponding control signal can be sent to instruct the display device 11 to open the on-screen keypad for inputting characters.

FIG. 4 is a schematic diagram of a first type of operation interface according to an embodiment of the present invention. Please refer to FIG. 4 , in an embodiment, the first type of operation interface includes an operation interface 41 . The operation interface 41 can be, for example, an operation interface provided on a mobile device side for supporting web browsing or file management (such as a mobile version of Google Chrome application program). It should be noted that the configuration of the operable objects in the operation interface 41 may be different from the configuration of the operable objects in the operation interface 31 of FIG. 3 . For example, a plurality of operable objects 310 , 341 - 344 , 410 and 421 - 424 may be presented in the operation interface 41 . The operable objects 310 , 341 - 344 are the same as or similar to the elements with the same numbers in FIG. 3 . In one embodiment, the operation interface 41 may be presented according to the type of the application program (ie, the first application program) running on the display device 11 in FIG. 1 in response to the user clicking the operable object 310 .

The operable object 410 includes a virtual touchpad, which is used to detect the user's touch operation gestures. For example, when a user's touch operation gesture on the virtual touchpad is detected, a corresponding control signal can be sent. In particular, the control signal can instruct the display device 11 to perform various touch operations such as screen sliding, screen zooming, and cursor movement on the presented API of the first type of application program. The touch operation behavior indicated by operating the operable object 410 or driven to be executed by the display device 11 may be the same as or different from the touch operation behavior instructed or driven to be executed by the display device 11 by operating the operable object 320 in FIG. 3 .

The operable objects 421-424 include a plurality of functional objects. For example, when it is detected that the user clicks on the operable object 421 , the corresponding control signal sent may instruct the display device 11 to zoom in on the presented API of the first type of application. For example, when it is detected that the user clicks on the operable object 422 , a corresponding control signal may be sent to instruct the display device 11 to zoom out the displayed API of the first type of application. For example, when it is detected that the user clicks on the operable object 423, the corresponding control signal sent can instruct the display device 11 to copy the specific content (such as the selected text content) in the API of the first type of application program presented. ). For example, when it is detected that the user clicks on the operable object 424, a corresponding control signal may be sent to instruct the display device 11 to paste the copied content.

FIG. 5 is a schematic diagram of a second type of operation interface according to an embodiment of the present invention. Please refer to FIG. 5 , in an embodiment, the second type of operation interface includes an operation interface 51 . The operation interface 51 can be, for example, an operation interface on a mobile device provided for an application program that supports playing multimedia images (such as a TV version of the Netflix application program). It should be noted that the configuration of the operable objects in the operation interface 51 may be different from the configuration of the operable objects in the operation interface 31 of FIG. 3 and the operation interface 41 of FIG. 4 . For example, a plurality of operable objects 310 , 341 - 344 and 511 - 515 can be presented in the operation interface 51 . The operable objects 310 , 341 - 344 are the same as or similar to the elements with the same numbers in FIG. 3 . In one embodiment, the operation interface 51 may be presented according to the type of the application program (ie, the first application program) running on the display device 11 in FIG. 1 in response to the user clicking the operable object 310 .

The operable objects 511-515 include a plurality of virtual button objects. These virtual button objects can be used to simulate the object switching and object selection functions of a traditional physical remote controller. For example, when it is detected that the user clicks on the operable objects 511-514, a corresponding control signal can be sent. In particular, these control signals can instruct the display device 11 to perform object switching operations on multiple objects in the presented API of the first type of application program (for example, switch the object to be selected up, down, left, and right). ). For example, when it is detected that the user clicks on the operable object 515, the corresponding control signal sent may instruct the display device 11 to perform an object selection operation on an object in the presented API of the first type of application program. (for example, execute the function of the currently selected object).

It should be noted that, according to the embodiments of FIG. 3 and FIG. 4, at least some of the objects presented in the first type of operation interface can be based on the type of the application currently running on the display device 11 of FIG. program, mobile web browser, or other type of mobile application) and dynamically configured. In this way, the first type of operation interface presented on the mobile device (ie, the mobile device 12 in FIG. 1 ) can more appropriately provide operation functions (such as touch operation functions) corresponding to the application currently running on the display device 11 .

In addition, compared with the embodiments of FIG. 3 and FIG. 4 , in the embodiment of FIG. 5 , the second type of operation interface can also be based on the type of the application program currently running on the display device 11 (such as a TV version of a multimedia player program) In addition to configuration, at least some of the objects provided in the second type of operation interface can also be used to simulate physical buttons on a traditional physical remote control. In this way, when these objects in the second type of operation interface are triggered (for example, clicked), the corresponding control signal sent can instruct the display device 11 to perform a function similar to that triggered by pressing a physical button on a traditional physical remote control. Button operation function.

FIG. 6 is a flowchart of a display control method according to an embodiment of the present invention. Please refer to FIG. 6, in step S601, start the remote control program. In step S602, the remote control program detects the type of the application running on the display device. In step S603, according to the type of the application program, the remote control program presents one of the first type of operation interface and the second type of operation interface on the operation interface of the mobile device, so that the operation interface The provided operation functions meet the operation requirements of the application programming interface. In step S604, the remote control program detects the user's operation behavior on the operation interface. In step S605, the remote control program transmits a control signal to the display device according to the operation behavior, so that the display device operates the API according to the control signal.

However, each step in FIG. 6 has been described in detail above, and will not be repeated here. It should be noted that each step in FIG. 6 can be implemented as a plurality of program codes or circuits, which is not limited by the present invention. In addition, the method in FIG. 6 can be used in combination with the above embodiment, or can be used alone, which is not limited in the present invention.

To sum up, the display control method and display control system provided by the embodiments of the present invention can dynamically present an appropriate operation interface on the mobile device according to the type of the application currently running on the display device. The operation interface can be used by the user of the mobile device to instruct the display device to operate the presented application program interface by operating the operation interface, such as simulating touch operation behavior or performing traditional button operation behavior. In this way, the compatibility and convenience of using the mobile device to operate and control different types of application programs displayed on the remote display device can be improved.

10: Display control system 11: Display device 12: Mobile Devices 21: Processor 22: storage circuit 23: Input/Output Interface 201: Remote control program 31, 41, 51: Operation interface 310, 320, 330, 341~344, 421~424, 511~515: operable objects S601~S605: steps

FIG. 1 is a schematic diagram of a display control system according to an embodiment of the present invention. FIG. 2 is a functional block diagram of a mobile device according to an embodiment of the present invention. FIG. 3 is a schematic diagram of a first type of operation interface according to an embodiment of the present invention. FIG. 4 is a schematic diagram of a first type of operation interface according to an embodiment of the present invention. FIG. 5 is a schematic diagram of a second type of operation interface according to an embodiment of the present invention. FIG. 6 is a flowchart of a display control method according to an embodiment of the present invention.

S601~S605: steps

Claims (10)

A display control method for a mobile device, wherein the mobile device is wirelessly connected to a display device, the display device runs an application program and presents an API of the application program on a display interface, and the display control method includes : start a remote control program; send a request to the display device by the remote control program to request the identification information of the application program; receive the identification information of the application program from the display device by the remote control program; The remote control program detects the type of the application program according to the identification information; according to the type of the application program, the remote control program presents one of a first type of operation interface and a second type of operation interface On an operation interface of the mobile device, the operation function provided by the operation interface meets an operation requirement of the application program interface; the remote control program detects an operation behavior of a user on the operation interface; and by The remote control program sends a control signal to the display device according to the operation behavior, so that the display device operates the API according to the control signal. The display control method as described in claim item 1, wherein the type of the application program includes a first-type application program and a second-type application program, and the remote control The process program sends the control signal to the display device according to the operation behavior, so that the display device operates the API according to the control signal. The step includes: in response to the type of the application being the first type of application, sending a A first control signal is sent to the display device, so that the display device operates the API according to a touch operation behavior; and in response to the type of the application program being the second type of application program, sending a second control signal to the display device The display device enables the display device to operate the API according to a button operation behavior. The display control method as described in claim 2, wherein the touch operation behavior includes at least one of a screen slide, a screen zoom, and a cursor movement for the application programming interface, and the button operation behavior includes for the At least one of an object switching and an object selection of the API. The display control method according to claim 2, wherein the first type of application program includes a mobile version of the application program. The display control method as described in claim 2, further comprising: adjusting at least one of the first-type operation interfaces presented on the mobile device in response to different types of the first-type application programs running on the display device. The configuration of the operation object. A display control system, comprising: a display device for running an application program and presenting an API of the application program on a display interface; and a mobile device wirelessly connected to the display device, Wherein the mobile device is used to: start a remote control program; send a request to the display device by the remote control program to request the identification information of the application program; receive the application from the display device by the remote control program The identification information of the program; the remote control program detects the type of the application program according to the identification information; according to the type of the application program, the remote control program combines a first type of operation interface and a second type of operation One of the interfaces is presented on an operation interface of the mobile device, so that the operation function provided by the operation interface meets an operation requirement of the application program interface; the remote control program detects a user's operation interface an operation behavior; and the remote control program sends a control signal to the display device according to the operation behavior, so that the display device operates the API according to the control signal. The display control system as described in claim 6, wherein the type of the application program includes a first-type application program and a second-type application program, and the remote control program sends the control signal to the display according to the operation behavior device, so that the display device operates the API according to the control signal, including: in response to the type of the application being the first type of application, sending a first control signal to the display device, so that the display The device responds to a touch operation behavior operating the API; and in response to the type of the application being the second type of application, sending a second control signal to the display device, so that the display device operates the API according to a button operation behavior. The display control system as described in claim 7, wherein the touch operation behavior includes at least one of a screen slide, a screen zoom, and a cursor movement for the application programming interface, and the button operation behavior includes for the At least one of an object switching and an object selection of the API. The display control system according to claim 7, wherein the first type of application program includes a mobile version of the application program. The display control system as described in claim 7, wherein the mobile device is further used to: adjust the display displayed on the mobile device by the remote control program in response to the different types of the first type of application programs running on the display device The configuration of at least one operable object in the first type of operation interface.

Images