JP6785524B2 - Display device, display method and display program - Google Patents

Description

The present disclosure relates to display devices, display methods and display programs for linking native applications with web applications.

By implementing the application as a web application that can be executed on a web browser, it is not necessary to install dedicated software, and convenience is improved. Non-Patent Document 1 describes a method of implementing a remote collaborative work system as a web application. By accessing a predetermined URL from the user's terminal with a web browser, a shared workspace in which functions for viewing / editing documents, images, videos, etc. are arranged can be displayed on the web browser. Changes in the arrangement and contents of documents, etc. are reflected in real time on the terminals of all users participating in the collaborative work.

Web applications have the advantage that they do not require the installation of dedicated software, but the functions that can be realized by web applications are limited. For example, it is currently not possible to view the contents of a PostScript file on a web browser. In order to handle files that are not supported by a web browser on a web application, it is necessary to develop such a function as a web application, which requires a great deal of effort. On the other hand, some native applications implemented in C ++ or the like can display the contents of a PostScript format file.

T. Marlinan, J.M. Aurisano, A. Nishimoto, K.K. Baradwaj, V.I. Matevitsi, L.M. Renambot, L. et al. Long, A. Johnson, and J. Leeg, "SAGE2: A New Application for Data Collaboration Collaboration Using Scalable Resolution Shared Displays", CollaborateCom. 2014.

In order to solve the above problems, the present disclosure aims to link a native application having a function that cannot be realized by a web application with the web application.

The present disclosure enables the function extension of a web application by linking the web application executed on the web browser with the native application. Specifically, a native application with functions that cannot be realized by a web application is displayed in the same position and size as the web application, and the operation from the user is transmitted.

Specifically, the display device according to the present disclosure is
The native application part that executes native applications that cannot be executed by web applications,
The web browser section that displays the web application and the web browser section that displays the web application so that the area for displaying the native application is transparent in the display area of the web application.
A terminal control unit that starts the native application unit in response to a request from the web browser unit and superimposes the display of the native application on the area made transparent by the web browser unit.
With
When the web browser unit receives the user input in the display area of the native application, the web browser unit detects the position where the user input is received and the operation content, and detects the position and the operation content.
The terminal control unit generates the position and operation content received by the user in the native application in the native application unit using the API of the OS of the own device .

When the web browser unit acquires a request for using a native application in the display area of the web application, the web browser unit transmits information including the native application and the display area requested to be used to the server device, and the information is transmitted from the server device. Upon receipt, the display area is made transparent.

Specifically, the display method according to the present disclosure is a display method executed by a display device.
The steps that the web browser section displays the web application,
The area for the web browser section to display the native application is made transparent in the display area of the web application, the terminal control section starts the native application section that executes the native application, and the web browser section makes the display of the native application transparent. Steps to overlap with the area
Have a,
When the web browser unit receives the user input in the display area of the native application, the web browser unit detects the position where the user input is received and the operation content, and detects the position and the operation content.
The terminal control unit generates the position and operation content received by the user in the native application in the native application unit using the API of the OS of the own device .

The display program according to the present disclosure causes the computer to function as each functional unit provided in the display device according to the present disclosure, and causes the computer to execute each step provided in the display method according to the present disclosure. The display program according to the present disclosure can be recorded on a recording medium or provided through a network.

According to the present disclosure, a native application having functions that cannot be realized by a web application can be linked with the web application.

The first system configuration example of this disclosure is shown. A second system configuration example of the present disclosure is shown. A first system configuration example when there are a plurality of display devices is shown. A second system configuration example when there are a plurality of display devices is shown. An example of a typical screen configuration in a display device is shown. An example of the format of the OPEN request is shown. An example of the management table is shown. An example of the setting information of the terminal control unit is shown. An example of the Move request format is shown. An example of the relative coordinates of the user input event is shown. An example of notification of a user input event is shown. An example of the screen configuration of the display device according to the first embodiment is shown. An example of the screen configuration of the display device according to the second embodiment is shown.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The present disclosure is not limited to the embodiments shown below. Examples of these implementations are merely examples, and the present disclosure can be implemented in various modifications and improvements based on the knowledge of those skilled in the art. In this specification and drawings, the components having the same reference numerals shall indicate the same components.

A basic configuration example of the present disclosure is shown in FIG. The system according to the present disclosure includes a server device 91 and a display device 92. The server device 91 functions as a web server. The display device 92 includes a web browser unit 21, a native application unit 23, and a terminal control unit 22. The server device 91 and the web browser unit 21 of the display device 92 can communicate with each other via a network. The apparatus of the present disclosure may be realized by making a computer function as each functional unit.

The web browser unit 21 downloads and executes a web application from the server device 91. As a result, the web browser unit 21 can display the web page on the display device 92. For the communication here, HTTP (Hypertext Transfer Protocol) may be used. Further, WebSocket may be used to enhance the real-time property of communication. A web application is generally an application whose behavior is described in a script such as HTML, CSS, or Javascript (registered trademark), but if it can be executed on the web browser unit 21, the behavior is described in another format. It may be described.

The native application unit 23 executes an application having a function that cannot be realized by a web application. You can use the existing native application as it is. There may be a display device 92 that uses only the web browser unit 21 without using the native application unit 23. Further, the number of native applications used by the native application unit 23 may be plural.

Communication between the web browser unit 21 and the terminal control unit 22 may go through the server device 91 as shown in FIG. Alternatively, as shown in FIGS. 2 and 4, the terminal control unit 22 may have a server function corresponding to HTTP or WebSocket and directly communicate with the web browser unit 21.

Further, there may be a plurality of display devices 92. In that case, the same screen display is performed on all the display devices 92. When there are a plurality of display devices 92 in the configuration of FIG. 1, the web browser unit 21 and the terminal control unit 22 of each display device 92 communicate with the server device 91 via the network as shown in FIG. When there are a plurality of display devices 92 in the configuration of FIG. 2, one web browser unit 21 directly communicates with the terminal control unit 22 provided in all the display devices 92 as shown in FIG. Hereinafter, the communication between the web browser unit 21 and the terminal control unit 22 includes both cases. Further, the communication between the web browser units 21 shall be via the server device 91.

FIG. 5 shows a typical screen configuration of the present disclosure in the display device 92. For example Windows using a computer equipped with (R) OS as the display device, by displaying a native application A _N Align below the web application A _W on the desktop, displayed as integrated applications To do. In the area of on the web application A _W which overlaps with the display area of the native application A _N, clearing the background. A transparent area on the web application A _W, the position and alignment of the size of the native application A _N terminal control unit 22 manages.

Procedure before showing native application A _N is as follows. Web applications need to be written to behave accordingly.
First, the web browser unit 21 accesses a predetermined URL and downloads the web application. When a user requests a use of native applications A _N on the web application, the web browser 21 sends an OPEN request to the server device 91. The server device 91 transmits the received OPEN request to the web browser unit 21 of each display device 92.

For the OPEN request, Javascript (registered trademark) Object Notification (JSON) in the format shown in FIG. 6 may be used. The OPEN request includes an identification number (“app”) indicating the function to be used and a position (“position”) to be displayed. In the example of FIG. 6, "position" includes the start point horizontal position coordinate x, the start point vertical position coordinate y, the horizontal size w, and the vertical size h in this order.

The web browser unit 21 that has received the OPEN request makes the background of the area specified by "position" transparent. The area to be made transparent is defined for each function identification number. Further, at this time, the web browser unit 21 adds an entry to the management table as shown in FIG. 7 and holds layer information for expressing the position and size, and the upper and lower overlap. Here, it is assumed that the smaller the number is, the higher the layer is, and the value of the layer at the top is 1. The timing of updating the layer value depends on the implementation of the web application. The layer value may be set to 1 when a new native application is placed or when the position / size is changed. With such a mechanism, the web browser unit 21 grasps the hierarchical relationship between the native application application 1 and the native application application 2.

In order to operate the native application, the OPEN request transmitted by the web browser unit 21 is transmitted to the terminal control unit 22. Terminal controller 22 which has received the OPEN request is arranged to overlap underneath the transparent area native application _{A N} by using the OS of API (Application Programming Interface). The API is, for example, SetWindowPos in Windows®. The terminal control unit 22 includes a pre-function identification number "app", has information associating the path of native application _{A N} ( "path") in the display device 92, a Included "app" to OPEN request, which to determine whether to start the native application a _N. FIG. 8 shows an example of setting information given to the terminal control unit 22 for association.

Even when changing the position and size of the display of the native application A _N on the web application similarly sends a MOVE request including the changes from the web browser 21 via the server apparatus 91, and the transparent region of the native application A _N to match the position and size of the web application a _W. The format shown in FIG. 9 may be used for the MOVE request.

The transmission path of the OPEN request to the terminal control unit 22 is different in each configuration of FIGS. 1 to 4. For example, in the case of the configurations of FIGS. 1 and 3, the server device 91 transmits an OPEN request to the terminal control unit 22 of each display device 92. For example, in the case of the configurations of FIGS. 2 and 4, the web browser unit 21 that transmits the OPEN request also sends the OPEN request to the terminal control unit 22 of its own device. Then, the web browser unit 21 that has received the OPEN request from the server device 91 sends the received OPEN request to the terminal control unit 22 of its own device. For example, in the case of the configuration of FIG. 4, the web browser unit 21 that transmits the OPEN request can also send the OPEN request to the terminal control unit 22 of the other display device 92.

A display device 92 to use the native application A _N, if the display device 92 is not used are mixed, for example by including a flag in the URL, you can be distinguished them. When the URL of the web application is "http://foo.bar/index.html", the URL when the native application is not used is "http://foo.bar/index.html?native=file". The URL of the case may be "http://foo.bar/index.html? Native = true". By the above method, without modifying the native application A _N, it is possible to screen that links to the web application A _W and a native application A _N.

However, only overlapped the display, when the user attempts an operation such as a mouse click to the native application A _N, web application A _W of the overlying will receive the operation. Therefore, this embodiment has a mechanism for transmitting a user input event to the transparent region of the web application A _W, the native application A _N.

As shown in FIG. 10, the coordinates of the user input position P _I on the Web browser B _W user input event has occurred and (a, b). The web browser unit 21 detects a user input in the display area of the web application A _W, to determine coordinates (a, b) with reference to the table of FIG. 7 is included in the area of any function. If there are multiple applicable entries, select the entry with the smaller layer value.

The web browser unit 21 transmits a user input event including an identification number (“app”) and an operation content (“input”) of the selected function to the terminal control unit 22 provided in all the display devices 92. Coordinates transmitted at this time is obtained by converting the relative coordinates in the display area of the native application A _N. That is, the coordinates (a, b) input position _{P I} is a display area of the native application _{A N,} when the start position of the area of the native application _{A N} coordinates (X, Y) and the coordinate (a-X, b-Y) is sent. The JSON format as shown in FIG. 11 may be used for the notification of the user input event.

The terminal control unit 22 determines a native application that passes an event from "app". Native application determined by the terminal control unit 22, relative coordinates (a-X, b-Y ) on the native application A _N using the API of the OS to generate an event.

(Embodiment 1)
FIG. 12 shows a configuration example of a signage that links the web application A _W and the native application A _N. In the present embodiment, in the software advertisement, the text or the like is displayed by the web application A _W , and the actual software is displayed as the native application A _N by superimposing it under the web application A _W.

In this embodiment, the web browser unit 21 downloads the web application. The native application may be stored in the display device 92 in advance. Alternatively, the web browser unit 21 may download it. The native application is stored in the display device 92 as the native application unit 23.

In the present embodiment, the display area of the native application A _N in the web application A _W is previously transparent set. When there is user input within the display area of the native application A _N, native application unit 23 generates an event on the native application A _N. Thus, a user having an interest in the software is able to check the actual operation to behave like a software button by entering a number in the frame that is displayed in the native application A _N.

(Embodiment 2)
In this embodiment, an example is shown in which a native application _AN displays a screen image of a workstation on which CAD software for designing a car operates, and a plurality of display devices 92 perform remote cooperative work while watching the screen image. It is shown in 13.

The display device 92-1 is a terminal of a user who participates from a mobile environment where only a narrow band network can be used, and the display device 92-1 is a terminal of a user who can use a high-performance terminal and a wideband network. Server device 91, with respect to these display devices 92-1 and 92-2, provides a common web application _{A W.} At this time, by using different native application units 23 for displaying the screen image of the workstation, services of different qualities are provided in the screen image of the workstation.

In order to distinguish between these user terminals, the web browser unit 21 provided in the display devices 92-1 and 92-2 accesses different URLs. The web browser section 21, documents, images, video, etc. to display the web application A _W as a shared work space for disposing the ability to view and edit, as well against either except screen video display function URL behave.

For the user of the display device 92-1 to participate from the mobile environment, in order not to generate the installation of the new software, to be able to perform all of the functions on the web application A _W. Therefore, when accessing from the URL for the mobile environment, the web application is described so as to display the screen image of the workstation by using the image transmission / reception mechanism that can be executed on the web browser. As such a mechanism for transmitting and receiving video, for example, there is WebRTC (Web Real-Time Communication). The quality of video transmission / reception by WebRTC is not so high, but convenience is improved by being able to participate in collaborative work without installing a native application in advance.

On the other hand, as in-house conference rooms, the high-performance terminal and the broadband network is available the display device 92-2, a native application unit 23 displays the screen image to the native application A _N. Therefore, the web application is described so as to start the native application for video transmission / reception using the method of the present disclosure when accessing from the URL for the high-performance terminal. The native application unit 23 is, for example, an application that can use a JPEG2000 codec that can obtain a high-definition image instead of requiring a wide band. In this way, by displaying a screen image to the native application A _N using native application section 23, effective use of computing resources and network bandwidth of the terminal can confirm the detail of the design.

(Effect of disclosure)
As described above, the present disclosure is by linking the web application A _W and native applications A _N running on the web browser, allowing extension of the web application A _W. Specifically, the native application A _N having a function that can not be achieved in the web application A _W, and displays the combined position and size as the web application A _W, realized by transmitting an operation from a user. Native application A _N is previously web application A _W and need not be those made to cooperate, it is possible to directly utilize existing native application A _N. Further, since the same screen can be displayed on a plurality of display devices, it can be applied to remote cooperative work. Furthermore, it becomes possible to provide services of different qualities even among users who participate in the same collaborative work.

This disclosure can be applied to the information and communication industry.

21: Web browser unit 22: Terminal control unit 23: Native application unit 91: Server device 92, 92-1, 92-2: Display device

Claims (4)

The native application part that executes native applications that cannot be executed by web applications,
The web browser section that displays the web application and the web browser section that displays the web application so that the area for displaying the native application is transparent in the display area of the web application.
A terminal control unit that starts the native application unit in response to a request from the web browser unit and superimposes the display of the native application on the area made transparent by the web browser unit.
With
The web browser section
When a user input is received in the display area of the native application, the position where the user input is received and the operation content are detected.
Do not change the display area of the web application before and after running the native application
After running the native application, display according to the web application in the display area of the web application,
In the display area of the native application , the terminal control unit generates the position and operation content of the user input in the native application in the native application unit using the API of the OS of the own device.
Display device. The web browser section
When a request to use a native application is acquired in the display area of a web application, information including the native application and the display area requested to be used is transmitted to the server device.
When the information is received from the server device, the display area is made transparent.
The display device according to claim 1. The steps that the web browser section displays the web application,
The area for the web browser section to display the native application is made transparent in the display area of the web application, the terminal control section starts the native application section that executes the native application, and the web browser section makes the display of the native application transparent. Steps to overlap with the area
Have,
When the web browser unit receives the user input in the display area of the native application, the web browser unit detects the position where the user input is received and the operation content, and detects the position and the operation content.
In the display area of the native application , the terminal control unit generates the position and the operation content received by the user in the native application in the native application unit using the API of the OS of the own device .
The web browser unit does not change the display area of the web application before and after the execution of the native application, and displays according to the web application in the display area of the web application after the execution of the native application.
Display method. A display program that causes a computer to function as each functional unit provided in the display device according to claim 1 or 2.

Images