WO2024029094A1 - Screen display system and screen display method - Google Patents

Abstract

A device (110) generates screen information indicating the control state of the device, and sends the generated screen information to a terminal (120). The terminal (120) generates a display screen based on the screen information received from the device (110), and displays the display screen on a display unit (121). When in an initial state, the device (110) shares with the terminal (120) initial display list information (K0), which includes a character string and imagery relating to GUI screen construction and to the arrangement of each constituent part. After the device (110) begins to operate, GUI screen information including the character string and the imagery corresponding to a prescribed state is selected from the initial display list information (K0), and is then sent to the terminal (120). On the basis of the received GUI screen information, the terminal (120) references the shared initial display list information (K0), and then performs screen display processing to display the operating state of the device (110) on the display unit (121).

Description

Screen display system and screen display method

The present invention relates to a screen display system and a screen display method that transfer screen information generated by a device to a terminal and display it.

By using a GUI (Graphic User Interface) screen, the user can easily and intuitively operate the device. By transferring the display data of the GUI screen between devices, for example, a user can easily operate the device by using a portable terminal and operating the GUI screen on the terminal.

As a conventional technique, there is a technique that reduces the amount of GUI data by storing only glyph data used for GUI display among the GUI data stored in the storage section of a terminal with memory capacity constraints ( For example, see Patent Document 1 below.) In addition, in remote desktop, when the screen image on the remote terminal screen is changed, there is a technology that sends the difference screen image, changed area information, and difference drawing area information to the local terminal to rebuild the screen ( For example, see Patent Document 2 below.)

Japanese Patent Application Publication No. 2008-117319 Japanese Patent Application Publication No. 2010-146504

Embedded devices that perform predetermined control can be easily controlled by displaying control details on a GUI screen, but some devices are configured to perform not only GUI screen generation processing but also drawing processing. In this case, the number of hardware parts for the embedded device increases, making it impossible to reduce costs. For example, in the case of a system configuration in which a terminal is communicatively connected to an embedded device, it is necessary to efficiently separate the functions of the GUI placed on the embedded device side and the GUI function placed on the terminal side. Further, in the conventional technology, image data to be displayed on a remote terminal is transmitted to a local terminal, so the amount of data to be transferred has increased.

In view of the above problems, an object of the present invention is to be able to transfer a display screen indicating the status of a device to a terminal without increasing the amount of data.

In order to achieve the above object, the screen display system of the present invention is a screen display system that transfers screen information from a device to a terminal, in which the device includes a control section that generates screen information indicating the control state of the device itself, and a screen display system that transfers screen information from a device to a terminal. a communication unit that transmits screen information generated by the control unit to the terminal, and the terminal includes a processing unit that generates a display screen based on the screen information received from the device, and a communication unit that transmits the screen information generated by the processing unit. a display unit that displays the display screen, and the control unit of the device, in an initial state, displays an initial display including character strings and images regarding GUI (Graphic User Interface) screen construction and arrangement of each component. List information is shared with the terminal, and after operation of the device, screen information including the character string and the image corresponding to a predetermined state is selected from the initial display list information and sent to the terminal. The processing unit of the terminal refers to the shared initial display list information based on the received screen information, and performs screen display processing on the operating state of the device.

Further, the device does not have a display section for the screen information within its own device, and the terminal refers to the shared initial display list information based on the received screen information and responds to the state of the device. It is characterized by performing screen display processing and displaying it on the display unit.

Further, the device assigns an identifier to each of the character string and image included in the initial display list information, and transmits the identifier to the terminal as the screen information, and the terminal transmits the identifier to the received screen information. Based on this, the shared initial display list information is referred to, and screen display processing corresponding to the state of the device is performed.

Further, the device is characterized in that the device transmits information regarding the shape, character string, display coordinates, and color of each component to the terminal as the initial display list information.

Further, the device updates the screen information in response to the change in the state, generates only information regarding the shape, text, display coordinates, and color of the component to be updated as the screen information, and transmits it to the terminal. It is characterized by

Further, the terminal includes an operation section, and transmits operation information from the operation section to the device, and the device performs control based on the operation information received from the terminal, and based on the control, the screen information and transmits the updated screen information to the terminal.

Further, the terminal is characterized in that it is communicatively connected to a plurality of the devices, displays display information of the communicatively connected devices on the display section, and controls the devices by operating the operation section. .

Furthermore, the device is characterized in that it is a built-in device that performs predetermined control with limited use.

Further, the screen display method of the present invention is a screen display method that transfers screen information from a device to a terminal, in which the device generates screen information indicating a control state of its own device, and transmits the generated screen information to the terminal. The terminal displays a display screen based on the screen information received from the device, and the device, in an initial state, displays character strings and Initial display list information including images is shared with the terminal, and after operation of the device, screen information including the character string and the image corresponding to a predetermined state is selected from the initial display list information. and transmits the information to the terminal, and the terminal refers to the shared initial display list information based on the received screen information and performs screen display processing of the operating state of the device.

According to the above configuration, the screen display system shares the initial display list information with the terminal when the device is in the initial state. After the device is in operation, by selecting screen information corresponding to a predetermined state from the initial display list information and transmitting it to the terminal, the terminal can display a display screen indicating the state of the device on the display unit. In this way, by sharing the initial display list information between the device and the terminal in advance, it is possible to reduce the amount of data related to screen information displayed on the terminal after the device is in operation.

The GUI screen information sent from the device to the terminal consists of the shape, text characters, coordinates, colors, etc. of each object, and is not image data itself and has a small amount of data. As a result, GUI screen data can be transferred from the device to the terminal with a small amount of data, and the processing load on the device can be reduced. Further, when updating the GUI screen, the device transmits screen information such as only updated objects, shapes, text characters, coordinates, colors, etc. to the terminal. This makes it possible to further reduce the amount of data needed to update the GUI screen and transfer it from the device to the terminal with a smaller amount of data.

The device is, for example, a built-in device that performs predetermined control for a limited purpose and does not have a display section, and the user can grasp the control status of the device from the display screen on the terminal and control the operation section. Devices can be controlled by operation. This eliminates the need for screen drawing in the device, reduces the processing load, and reduces the cost of the device.

According to the present invention, it is possible to transfer the display screen indicating the status of the device to the terminal without increasing the amount of data.

FIG. 1 is a diagram showing a configuration example of a screen display system according to an embodiment. FIG. 2 is a diagram showing the functions of the screen display system according to the embodiment. FIG. 3 is a block diagram showing an example of the hardware configuration of the screen display system according to the embodiment. FIG. 4 is a diagram illustrating the transfer state of display data when the screen display system starts up the system. FIG. 5 is a diagram illustrating the state of transfer of a list of character strings and images when the screen display system starts up the system. FIG. 6 is a diagram illustrating the updated state of character strings and images after the screen display system starts up the system. FIG. 7 is a diagram illustrating display and display updating by the screen display system. (Part 1) FIG. 8 is a diagram illustrating display and display updating by the screen display system. (Part 2) FIG. 9 is a flowchart showing display and display update processing by the screen display system. (Part 1) FIG. 10 is a flowchart showing display and display update processing by the screen display system. (Part 2) FIG. 11 is a flowchart showing the transfer state during display and display update according to the conventional technology. (Part 1) FIG. 12 is a flowchart showing the transfer state during display and display update according to the prior art. (Part 2) FIG. 13 is an explanatory diagram showing an example of a screen displayed on a terminal by the screen display system according to the embodiment. (Part 1) FIG. 14 is an explanatory diagram showing an example of a screen display on a terminal by the screen display system according to the embodiment. (Part 2) FIG. 15 is a diagram showing an example of the configuration of a conventional embedded device. FIG. 16 is a diagram showing other functions of the screen display system according to the embodiment. FIG. 17 is a diagram illustrating the creation state of a conventional GUI application. FIG. 18 is a diagram illustrating an example of a GUI application update process of the screen display system according to the embodiment. FIG. 19 is an explanatory diagram of a comparison of development man-hours for different types of embedded devices. (Part 1) FIG. 20 is an explanatory diagram of a comparison of development man-hours for different types of embedded devices. (Part 2) FIG. 21 is an explanatory diagram of a comparison of development man-hours for different types of embedded devices. (Part 3)

(Embodiment)
DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a screen display system and a screen display method according to the present invention will be described in detail below with reference to the accompanying drawings.

FIG. 1 is a diagram showing a configuration example of a screen display system according to an embodiment. Screen display system 100 includes an embedded device (local device) 110 and a terminal 120 (remote device).

The embedded device 110 is, for example, a device for a specific purpose with limited use (operation), such as a remote control, a digital camera, an IC recorder, an in-vehicle device, a medical device, a surveillance camera, or an intrusion detector. The terminal 120 is a device (smart device) that a user can carry and move freely, such as a smartphone or a tablet. The terminal 120 and the embedded device 110 are communicably connected, for example, by wireless communication such as Wi-Fi (registered trademark).

Sensors A and B (111), LED1 (112), etc. are connected to the embedded device 110, and a control unit (not shown) generates GUI screen information. It also includes a communication unit that transmits GUI screen information generated by the control unit to the terminal 120. For example, the control unit of the embedded device 110 performs predetermined processing based on the detection of sensors A and B (111) connected to the embedded device 110, and displays the processing results on LED1 (112) or the like. Further, the control unit of the embedded device 110 performs a predetermined control operation to control the embedded device 110 based on the operation information of the terminal 120.

In this control operation, the control unit of the embedded device 110 controls the control state based on the values detected by the sensors A and B (111), information on user operations, etc., for example, the GUI screen regarding turning on/off of the LED1 (112). Generate information.

Here, the embedded device 110 does not display the generated GUI screen information within its own device, but transmits it to the terminal 120 and displays it on the terminal 120. That is, the embedded device 110 does not have a display section.

The terminal 120 has a communication unit that receives GUI screen information transmitted from the embedded device 110, a control unit that controls display of the GUI screen information, and a display unit 121 that displays the GUI screen information. The display unit 121 can also receive user operations via a touch panel or the like, and the control unit transmits operation information based on the input of the user operation to the embedded device 110.

The screen display system 100 described above transmits the GUI screen information generated by the embedded device 110 to the terminal 120, and displays the GUI screen on the display unit 121 of the terminal 120. That is, a GUI screen controlled by the embedded device 110 is displayed on the display unit 121 of the terminal 120.

FIG. 2 is a diagram showing the functions of the screen display system according to the embodiment. The internal functions of the embedded device 110 and the terminal 120 are shown respectively. In FIG. 2, functions related to GUI screen display are mainly described.

The embedded device 110 has a GUI application 201 that is executed by the control unit. The GUI application 201 includes a screen construction processing unit 211 that constructs a GUI screen. The screen construction processing unit 211 generates component placement information k of components included in the constructed GUI screen, and transmits it to the terminal 120 as GUI screen information K.

GUI screen information K (component placement information k) includes components of the GUI screen displayed on the display unit 121 of the terminal 120, for example, sensors A and B shown on the GUI screen displayed on the display unit 121 of the terminal 120 in FIG. (111), includes the lighting/unlighting arrangement position of LED1 (112), and includes, for example, coordinate information on the GUI screen.

The terminal 120 includes a screen drawing processing section 220. The screen drawing processing unit 220 performs screen drawing processing based on the GUI screen information K received from the embedded device 110 and displays and outputs the GUI screen on the display unit 121.

When the values of sensors A and B (111) connected to the embedded device 110 change, the sensors corresponding to sensors A and B (111) are displayed on the GUI screen displayed on the display unit 121 of the terminal 120. The displayed value changes. Similarly, the on/off state of LED1 (112) in the embedded device 110 is displayed on the display unit 121 of the terminal 120 (see FIG. 1 for the display state).

Furthermore, by operating the operating unit 221 provided on the terminal 120, the operating information S can be transmitted to the embedded device 110. For example, turning on/off the LED1 (112) on the GUI screen displayed on the display unit 121 shown in FIG. 1 can be changed by operating a button. At this time, the GUI application 201 provided in the embedded device 110 performs display control to update the GUI screen based on the operation information S of the terminal 120.

FIG. 3 is a block diagram showing an example of the hardware configuration of the screen display system according to the embodiment. Examples of hardware configurations of the above-described embedded device 110 and terminal 120 are shown.

The embedded device 110 includes a control unit (CPU) 301, a read-only memory (ROM) 302, a random access memory (RAM) 303, a storage unit 304 such as a semiconductor memory or a disk drive, and a communication interface (I/ F) 305. These CPU 301 to communication I/F 305 are connected to each other by a bus 308.

The CPU 301 is an arithmetic processing unit that controls the entire embedded device 110. Then, the CPU 301 executes the GUI application 201, generates the above-mentioned GUI screen, and transfers it to the terminal 120. The ROM 302 is a nonvolatile memory that stores programs for the GUI application 201 and the like. The RAM 303 is a volatile memory used as a work area when the CPU 301 executes arithmetic processing of the GUI application 201 program. The storage unit 304 can store and hold data used to execute the functions of the embedded device 110, other applications, and the like.

The communication interface 305 serves as a network connection interface with the terminal 120 and controls the input and output of information to be communicated. Specifically, the communication interface (I/F) 305 is connected to a network 310 such as a wireless/wired Local Area Network (LAN), Wide Area Network (WAN), or the Internet, and allows the embedded device 110 to communicate with the terminal 120. Connecting.

In the embedded device 110, the CPU 301 executes a program of the GUI application 201 to generate GUI screen information K (component placement information k) for displaying the GUI screen, and sends the generated GUI screen information K to the terminal 120. It has the function of The function of transmitting the GUI screen information K to the terminal 120 and receiving the operation information S from the terminal 120 can be realized by the communication I/F 305 described above.

Further, the terminal 120 includes a control unit (CPU) 311, a ROM 312, a RAM 313, a storage unit 314 such as a semiconductor memory, a communication I/F 315, a touch panel 316, and a display 317. These CPU 311 to display 317 are connected to each other by a bus 318.

Additionally, the touch panel 316 has buttons for operating the embedded device 110. The touch panel 316 corresponds to the operation unit 221 in FIG. The display 317 is a display device that displays a GUI screen based on program execution by the CPU 311, and corresponds to the display unit 121 in FIGS. 1 and 2. For example, a thin film transistor (TFT) liquid crystal display, a plasma display, an organic EL display, or the like can be used as the display 317. A touch panel 316 is arranged on the display 317, and by operating a button portion on the GUI screen displayed on the display 317 on the touch panel 316, operation information S at the time of button operation can be output.

In this way, the terminal 120 can also be configured with hardware including the CPU 311, ROM 312, RAM 313, etc. described in FIG. The terminal 120 has a function of receiving GUI screen information K (component placement information k) from the embedded device 110 and drawing a GUI screen on the display 317 based on the received GUI screen information K. The screen drawing function can be realized by executing a program by the CPU 311 described above. Further, the function of receiving GUI screen information K from the embedded device 110 and transmitting the operation information S of the touch panel 316 to the embedded device 110 can be realized by the communication I/F 315 described above. As the terminal 120, for example, a general-purpose tablet, a smartphone, a portable PC, or the like can be used.

FIG. 4 is a diagram illustrating the transfer state of display data when the screen display system starts up the system. In the embodiment, in an initial state such as system startup, the local device (embedded device) 110 and remote device (terminal) 120 share static data related to display (images, character strings, etc.).

For example, when the system is started, the local device (embedded device) 110 stores information on a list of character strings and images (images, character strings, etc.) as static data for display stored in the storage unit 304. (initial display list information) is transferred all at once to the remote device (terminal) 120 via the network 310.

In addition, information such as a list of character strings and images (images, character strings, etc.) may be transferred from the cloud server 400 etc. to the terminal 120 as static data for display before the system is started. .

This static data for display (list of character strings and images) is held as the same data in the storage unit 304 of the embedded device 110 and the storage unit 314 of the terminal 120 at the time of system startup. .

As a result, static data for display held by the terminal 120 at the time of system startup can be held on the embedded device 110 side as well. The same static data for display held by both the terminal 120 and the embedded device 110 at the time of system startup is GUI screen information at the time of system startup, and is assumed to be initial display list information K0.

FIG. 5 is a diagram illustrating the state of transfer of a list of character strings and images when the screen display system starts up the system. FIG. 5 shows an example of a list 500 of character strings and images corresponding to the initial display list information K0 at the time of system startup. FIG. 5 shows an example in which a list 500 of character strings and images of the initial display list set in the initial display list information K0 is transferred from the embedded device 110 to the terminal 120 via the network at the time of system startup. shows.

The character string and image list 500 includes a character string list 510 and an image list 520. The list of character strings and images 500 is, for example, information on a list of character strings and a list of images that can be set in the embedded device 110. The information is not limited to this, and may be information on a list of character strings and a list of images that are set in the embedded device 110 at the time of system startup.

The example list of character strings and images 500 shown in FIG. 5 is a list of character strings and images that are set in the embedded device 110 at the time of system startup. In the embedded device 110, the setting states of some character strings and images are illustrated. Here, since the embedded device 110 does not have a display section, the character string "ABC" and the image "star (★) shape" are not actually displayed on the embedded device 110 side in FIG.

The embedded device 110, for example, stores a list of various character strings and images necessary for operation in advance.

In the character string list 510, the embedded device 110 holds information on character strings by number (record). For example, number 1 is the character string "aaa" and number 2 is the character string "ABC".

In the image list 520, the embedded device 110 holds image information by number (record). For example, number 1 is an image "star (★) shape" and number 2 is an image "triangle (▲) shape."

The terminal 120 displays the image information set on the embedded device 110 side on the display unit 121 of the terminal 120 by transferring the initial display list information K0 consisting of the information of the character string and image list 500 from the embedded device 110. can do.

Note that the numerical value "100" on the embedded device 110 is, for example, a sensor value detected by the embedded device 110 using a sensor or the like after the system is started. After the system starts, which will be described below, the embedded device 110 sequentially transfers sensor values detected over time to the terminal 120.

Further, due to functional expansion of the embedded device 110, character strings and images handled by the embedded device 110, that is, character strings and images held by the embedded device 110, may be updated. In this case, the initial display list information K0 of character strings and images is updated for the embedded device 110. Then, the updated initial display list information K0 may be transferred all at once from the embedded device 110 or the cloud server 400 to the remote device (terminal) 120.

FIG. 6 is a diagram illustrating the updated state of character strings and images after the screen display system starts up the system. In the example of FIG. 6, at a certain time after the system is started, the character string "ABC" and the image "star (★) shape" are selected from the character string and image list 500 due to the operation transition of the embedded device 110. shall be.

The embedded device 110 displays character strings and images corresponding to changing display contents (changes in setting contents since they are not actually displayed) based on the passage of time (operation transition) after the system is started, into the acquired initial display. It is selected from list information K0 and notified to terminal 120 as GUI screen information K (component placement information k).

Here, the numerical value "100" on the embedded device 110 is a sensor value detected by the embedded device 110 using a sensor or the like after the system is started.

In the example shown in FIG. 6, it is assumed that the embedded device 110 selects the corresponding character string "ABC" and the image "star (★) shape" in the character string list 510 at a certain time. In this case, the embedded device 110 transfers the character string number “2” of the character string list 510 and the display image number “1” of the image list 520 to the terminal 120 as the GUI screen information K. Furthermore, the detected sensor value “100” is included in the GUI screen information K and transferred to the terminal 120.

The embedded device 110 includes information regarding the arrangement of characters and parts (for example, the arrangement and area (size) of images and character strings, and the overlapping state of images and characters and images) in the GUI screen information K. In the example of FIG. 6, the GUI screen information K includes position coordinates (x, y), size (w, h), and character string number "2" for the character string "ABC".

Furthermore, the GUI screen information K includes position coordinates (x, y), size (w, h), and display image number "1" for the image "star (★) shape". Further, the GUI screen information K includes the coordinates (x, y) of the position, the size (w, h), and the display value "100" for the sensor value "100".

In this way, the embedded device 110 notifies the terminal 120 of the GUI screen information K including the selected characters, images, and sensor values, so that the terminal 120 displays information on the screen selected by the embedded device 110 on the display. It can be displayed on 121.

Thereafter, the embedded device 110 notifies the terminal 120 of the updated GUI screen information K every time a character, image, or sensor value is changed. At this time, the embedded device 110 only needs to notify the numbers (identifiers) that identify the character strings and images to be updated regarding the character strings and images in the GUI screen information K, which can reduce the amount of data to be transferred. .

Furthermore, the terminal 120 has already acquired information regarding the images to be displayed by transferring the initial display list information K0 consisting of the information of the character string and image list 500 at the time of system startup. Therefore, the terminal 120 can display an image using only the character string to be updated and the image number, and can perform display processing easily. In this way, according to the embodiment, it is not necessary to sequentially transfer the entire data (pixel image) when updating an image, so the amount of data to be transferred can be reduced.

(GUI screen transfer process)
7 and 8 are diagrams illustrating display and display updating by the screen display system. FIG. 7 shows the state of the GUI screen update operation. For example, the embedded device 110 updates the GUI screen based on a change in the detection state of the sensor value. For example, the GUI screen of the embedded device 110 can be updated based on the operation by the terminal 120.

FIG. 7 shows an example in which the embedded device 110 updates the GUI screen by operating the terminal 120 displaying the GUI screen. Touch the updated location on the touch panel of the display unit 121 of the terminal 120. For example, assume that an operation is performed to change the display coordinate position (x, y) of a square (□)-shaped component. In this case, the terminal 120 touches a square (□)-shaped component on the GUI screen and performs a drag operation to change the displayed coordinate position (x, y) to update the updated coordinates (tx, ty). ) is transmitted to the embedded device 110. The terminal 120 can similarly transmit the operation information S for updating the shape, character string, position (coordinates), color, etc. to the embedded device 110 by performing similar operations on other component parts.

FIG. 8 shows the updated GUI screen transfer status. On the embedded device 110 side, a GUI screen updated by operations on the terminal 120 or the like is transferred to the terminal 120.

The embedded device 110 updates the GUI screen based on operations on the terminal 120 and the like. In the example shown in FIG. 8, information on the coordinate position (x', y') of the updated display is generated for a square (□)-shaped component. Furthermore, information after changing the color to yellow is generated for the triangular (△)-shaped component. Also, information on the changed character string "aiu" is generated.

Then, the embedded device 110 transmits only the updated (changed) component component information to the terminal 120 as component placement information k'. In the example of FIG. 6, the embedded device 110 uses information about the coordinate position (x', y') of the updated display of a square (□)-shaped component and the color of a triangular (△)-shaped component. The information after changing the color to yellow and the information on the changed character string "aiu" are transmitted to the terminal 120 as component placement information k'.

In this way, when changing (updating) the GUI screen, the embedded device 110 transmits only the component placement information k' updated on the GUI screen to the terminal 120. Here, the component arrangement information k' to be transmitted at the time of update may be information on updated numbers for character strings and images, as shown in FIG. As a result, the embedded device 110 does not need to transmit the image data of the entire GUI screen each time the GUI screen is updated, and only needs to transmit the minimum component placement information k' necessary for updating the display screen. The amount of transferred data can be reduced.

9 and 10 are flowcharts showing display and display update processing by the screen display system. It mainly shows the processing performed by the CPU 301 on the embedded device 110 side and the CPU 311 on the terminal 120 side.

FIG. 9 shows the state when the system is started. First, the embedded device 110 determines whether the system is started (step S901). For example, the embedded device 110 waits until its own device starts up or is communicatively connected to the terminal 120 (step S901: No), and when its own device starts up or is communicatively connected to the terminal 120 (step S901: No). S901: Yes), the following processing is performed.

Next, the embedded device 110 obtains a list of character strings for image display and a list of images (step S902). For example, the information on the character string list and image list is set by external input to the embedded device 110 via the communication I/F 305 or by operation on the device itself. Note that the embedded device 110 may store a predetermined character string list and image list information in a non-volatile state in advance.

The list of character strings and images set in step S902 may be the minimum number of characters and images used by the embedded device 110. For example, the character strings may be character strings "sensor A", "sensor B", "LED1", etc. for identifying sensors A, B (111) and LED1 (112) connected to the embedded device 110 shown in FIG. It may be only a limited string of characters. In addition, the images are "square (□) shape" and "circle ( Only limited images such as "〇) shape" or "star (★) shape" may be used. Further, color information may be added to each image.

Next, the embedded device 110 generates a list of the set character strings and images as initial display list information K0 at system startup (step S903). The embedded device 110 stores and holds the generated initial display list information K0 at system startup in the storage unit 304. Then, the embedded device 110 transfers the generated initial display list information K0 at system startup to the terminal 120 via the communication I/F 305 (step S904).

The terminal 120 stores the initial display list information K0 transferred from the embedded device 110 in the storage unit 314 (step S905).

This allows the embedded device 110 and the terminal 120 to share the same initial display list information K0.

FIG. 10 shows the state of the GUI screen when it is updated. When updating the GUI screen information K, the embedded device 110 transfers the updated GUI screen information K to the terminal 120. For example, when the embedded device 110 receives the operation information S of the terminal 120, it updates the GUI screen information K based on the operation information S.

In the example shown in FIG. 10, the coordinate position of the image of the triangular (Δ)-shaped component and the information on the character string have been updated by operating the terminal 120. Regarding the image of the triangular (Δ)-shaped component, the coordinate position has been changed from (x, y) to (x', y'). It is also assumed that the character string has been changed from "ABCDE" to "XYZ".

In this case, the embedded device 110 generates only the updated image and text information of the component as component placement information k' (step S1001). In the example of FIG. 10, the embedded device 110 sends information on the triangular (△) shape (no change), new coordinate position (x', y'), and color (for example, blue) information (no change). generate. Also, for the constituent parts of the character string, generate new character string "XYZ" information, display coordinate position (x, y) information (no change), and color (e.g. white) information (no change). do.

Then, the CPU 301 of the embedded device 110 transmits the component placement information k' of only the component parts that have been changed due to the update to the terminal 120 via the communication I/F 305 (step S1002).

The terminal 120 receives the component placement information k' transmitted from the embedded device 110 via the communication I/F 315 (step S1003). Then, the terminal 120 performs screen drawing processing for the received component placement information k (step S1004).

At this time, the terminal 120 retains the GUI screen information K up to that point, and performs the drawing process by changing only the display state of the component updated by the reception.

In other words, the terminal 120 renders the unupdated round (◯) and square (□) shaped components without changing them. For the updated triangular (△)-shaped component, the same triangular (△)-shaped display position as before the update is moved to a new coordinate position (x', y'), and the same color (for example, blue) as before the update is applied. Process the drawing with . Furthermore, for the constituent parts of the updated character string, a new character string "XYZ" is drawn at the same coordinate position (x, y) as before the update and in the same color (for example, white) as before the update.

In this way, the embedded device 110 transfers the GUI screen information K to the terminal 120 as the component placement information k consisting of updated data for each component, so the image data (pixel image) of the entire screen is transferred each time the image data (pixel image) of the entire screen is updated. No need to send. In the example of FIG. 10, only the component placement information k' of the component P that has been changed due to the update is generated and transferred from the embedded device 110 to the terminal 120, so the amount of data to be transferred can be reduced. Furthermore, regarding the GUI screen information K (updated component placement information k') to be transferred, as shown in FIG. 5, only the updated character string and image number information (identifier information) need to be transferred. , the amount of data to be transferred can be further reduced.

As a result, the embedded device 110 does not need to transmit the image data of the entire screen each time the GUI screen is updated, and only the minimum component placement information k' necessary for updating the display screen is transmitted. The amount of data transferred between the device 110 and the terminal 120 can be made even smaller than the component placement information k. Further, in the terminal 120 as well, the display unit 121 can be updated with a minimum amount of data, and the processing load of drawing can be reduced.

(Comparison example with conventional image data transfer)
FIGS. 11 and 12 are flowcharts showing transfer states during display and display updating according to the conventional technology.

FIG. 11 shows the state of transfer of the display screen from the device 1101 to the terminal 1111. The device 1101 creates display data G (step S1101), and performs drawing processing on the display screen on the device 1101 (step S1102). Furthermore, the created display data G is transmitted to the terminal 1111 (step S1103).

The terminal 1111 receives the display data G transmitted from the device 1101 (step S1104). Then, the terminal 1111 performs screen drawing processing on the received display data G (step S1105) and displays it on the display screen 1112.

FIG. 12 shows the updated GUI screen transfer status. When updating the display screen, the device 1101 transfers the updated display data G' to the terminal 1111.

As shown in FIG. 12, it is assumed that the display data G' has updated coordinate positions of triangular (Δ)-shaped components and character string information compared to the display data G before update. For the triangular (△)-shaped component, the coordinate position is changed from (x, y) to (x', y'). Also, the character string has been changed from "ABCDE" to "XYZ".

However, in the conventional technology, even when the device 1101 updates (changes) the display data G, as shown in FIG. 12, the screen drawing process is the same as that in FIG. 11. As a result, the device 1101 creates updated display data G' for all parts (step S1201), and performs drawing processing on the display screen on the device 1101 (step S1202). Furthermore, the generated display data G' is transmitted to the terminal 1111 (step S1203).

The terminal 1111 receives the display data G' transmitted from the device 1101 (step S1204). The terminal 1111 then performs screen drawing processing on the received display data G' (step S1205) and displays it on the display screen 1112.

Therefore, the display data G' generated when updating the display screen shown in FIG. 12 has the same data size as the display data G before update. That is, display data G and display data G' have a data amount corresponding to the image (all pixels) of the entire display screen 1112.

As described above, in the conventional technology, the image data of the entire screen is transmitted, so the data size of the display data G transferred from the device 1101 to the terminal 1111 is larger than that in the embodiment. Furthermore, since the image data of the entire screen is transmitted even when the GUI screen is updated, the data size of the display data G' transferred from the device 1101 to the terminal 1111 is the same as the display data G and is compared to the embodiment. It's big.

(Example of remote display of GUI screen of embodiment)
Various application examples for displaying the GUI screen of the screen display system (embedded device 110) described above on the terminal 120 will be described.

FIGS. 13 and 14 are explanatory diagrams showing examples of screen displays on a terminal by the screen display system according to the embodiment.

FIG. 13 shows an example in which GUI screen information K generated by an embedded device 110 in a certain factory 1300 is displayed on the display unit 121 of the terminal 120. The embedded equipment 110 is, for example, equipment that is part of a production line for parts produced in the factory 1300. By carrying the terminal 120, the user can operate the embedded device 110 by viewing the display unit 121 at a location away from the embedded device 110, without directly operating the embedded device 110 at the location of the embedded device 110.

For example, if the embedded device 110 is installed in a clean room, the terminal 120 can display the control status of the embedded device 110 even if it is located outside the clean room. Thereby, the embedded device 110 can be operated at a remote location from the embedded device 110 without being limited to the location where the embedded device 110 is installed.

In the embodiment, the user can operate the embedded device 110 by viewing the display section 121 of the terminal 120 even if the embedded device 110 is not provided with a display screen. In this case, since the built-in device 110 is not provided with a display section, costs can be reduced. Furthermore, by communicatively connecting a plurality of embedded devices 110 to a terminal 120, the embedded devices 110 can be switched while looking at the display section 121 of one terminal 120, and the switched embedded device 110 can perform Monitoring of operating conditions and the like can be displayed on the display unit 121 of the terminal 120.

FIG. 14 shows an example in which GUI screen information K generated by the embedded device 110 in a certain factory 1401 is displayed on the display unit 121 of the terminal 120 in another factory 1402. The embedded equipment 110 is, for example, equipment that is part of a production line for parts produced in the factory 1401. By operating the terminal 120 at another factory 1402, the user can operate the embedded device 110 while looking at the display unit 121 at a location away from the embedded device 110, without directly operating it at the location of the embedded device 110. can do.

For example, if the embedded device 110 is installed in a factory 1401 in another country, the GUI screen information K of the embedded device 110 can be displayed by operating the terminal 120 located at the factory 1402 in the home country. As a result, the embedded device 110 can be operated at a remote location of the embedded device 110 without being limited to the installation location of the embedded device 110, and, for example, the embedded device 110 can be remotely maintained using the terminal 120.

(Comparison between the conventional technology and the embedded device of the embodiment)
FIG. 15 is a diagram showing an example of the configuration of a conventional embedded device. A conventional embedded device 1501 having functions equivalent to those of the embodiment (see FIG. 2) is shown. The conventional embedded device 1501 has a display section 1541 and an operation section 1542 in addition to a sensor 1511, an LED 1512, and a GUI application 1521.

The GUI application 1521 includes a screen construction processing section 1531 and a screen drawing processing section 1532 in order to display a GUI screen on the display section 1541 based on the component placement information k.

In this way, the conventional embedded device 1501 has hardware such as an operation section 1542 such as a touch panel and a display section 1541 such as a liquid crystal screen, which increases the weight and size of the embedded device 1501. , component costs were high and it was difficult to reduce costs. In addition, since the screen drawing processing performed by the screen drawing processing unit 1532 has a processing load and memory usage is high, in order to execute the GUI application 1521, a high-performance control unit (a microcomputer such as a CPU) and a large-capacity Memory is required, making it difficult to reduce the cost of the embedded device 1501.

In contrast, in the embodiment, as shown in FIG. 2, the screen drawing processing section 220 and the display section 121 are provided on the terminal 120 side, and are not provided on the embedded device 110. The embedded device 110 transmits GUI screen information K (component placement information k) to the terminal 120. In this way, the cost of the embedded device 110 can be reduced by providing the hardware for screen drawing processing and display in a general-purpose smart device (terminal 120). Furthermore, the embedded device 110 can reduce the processing load and memory usage for screen drawing processing and display.

Further, according to the embodiment, it is possible to eliminate the waste of displaying the same GUI screen on the embedded device 110 and the terminal 120, and to view the GUI screen by displaying the GUI screen only on the terminal 120 carried by the user. It can be presented only to the user, and waste such as power consumption associated with GUI screen display on the embedded device 110 can be avoided.

Note that since the terminal 120 is originally equipped with hardware for screen drawing processing and display, the processing load and memory usage for screen drawing processing and display on the terminal 120 do not increase, and costs do not increase. . Furthermore, by providing the operation section on the terminal 120 side from the embedded device 110, the cost of the embedded device 110 can be further reduced.

FIG. 16 is a diagram showing other functions of the screen display system according to the embodiment. The example shown in FIG. 16 shows a configuration example in which one terminal 120 is communicatively connected to a plurality of embedded devices 110 (110A to 110C) of different systems (types).

In the embodiment, as described above, the hardware for screen drawing processing and display is not provided in the embedded device 110 but is provided in the terminal 120. In this system configuration, as shown in FIG. 16, by communicatively connecting one terminal 120 to a plurality of embedded devices 110 (110A to 110C), one terminal 120 can connect to a plurality of embedded devices 110 (110A to 110C). The GUI screens (110A to 110C) can now be displayed. Furthermore, one terminal 120 can control a plurality of embedded devices 110 (110A to 110C).

For example, by operating the terminal 120 to switch to one embedded device 110A and establish a communication connection, the GUI screen being controlled by the communication-connected embedded device 110A can be displayed on the display unit 121 of the terminal 120. The built-in device 110A connected for communication can be controlled by operating the operation unit 221 of the terminal 120.

According to the system configuration shown in FIG. 16, conventionally, for example, the three embedded devices 110 (110A to 110C) each include a screen drawing processing section 1532, a display section 1541, and an operation section 1542 (FIG. 15). ), and unnecessary GUI displays at the installation location of the embedded device 110 can be omitted, making it possible to significantly reduce the cost of the entire system.

FIG. 17 is a diagram illustrating the creation state of a conventional GUI application. Next, the conventional method and the embodiment will be compared from the viewpoint of GUI application creation. The embedded device 110 in FIG. 17 shows an example without a GUI screen display function.

In such a conventional embedded device 110, it was necessary to create a GUI application 1721 on the external computer 1710 side such as a PC for each configuration of the embedded device 110. In the GUI application 1721, as described above, the screen construction processing section 1731 generates component placement information k, and the screen drawing processing section 1732 performs drawing processing on the GUI screen to be displayed on the display section 1741. It also processes to transmit operation information from the operation unit 1742 to the embedded device 110.

In the above configuration, each time the system configuration (type) of the embedded device 110 differs, the types of GUI applications 1721 on the external computer 1710 side increase, and it is necessary to create a GUI application 1721 for each system configuration. As a result, the software structure and functions of the GUI application 1721 become complicated, the number of development steps increases, and it becomes difficult to reduce development costs.

On the other hand, according to the system configuration of the embodiment, as shown in FIG. 2, the embedded device 110 of each system configuration has the minimum Only this part is placed inside the embedded device 110. That is, the GUI application 201 is arranged in the embedded device 110, and the screen drawing processing section 220 and the display section 121 (and the operation section 221) are arranged in the terminal 120.

As a result, on the smart device (terminal 120) side, the screen drawing processing unit 220 can perform GUI application processing as a general-purpose process of "performing screen drawing based on component placement information k." As a result, one type of GUI application for the terminal 120 can be used even for embedded devices 110 having different configurations, and the number of development steps can be reduced. In other words, the screen drawing processing unit 220 of the terminal 120 only needs to have a single function of drawing the screen based on the component placement information k sent from the embedded device 110, and can be general-purposed and is more efficient than conventional technology. It will be possible to reduce development costs.

Furthermore, in the conventional system configuration shown in FIG. 17, it is necessary to change the GUI application 1721 (screen construction processing section 1731, screen drawing processing section 1732) due to an OS version upgrade of the external computer 1710, etc. In this case, it is necessary to modify and retest all the individual GUI applications 1721, resulting in an increase in development man-hours.

In contrast, according to the system configuration of the embodiment (see FIG. 2), one type of GUI application (screen drawing processing unit 220, etc.) of the terminal 120 can be used even for embedded devices 110 with different configurations. Therefore, in the embodiment, it is only necessary to change and retest the GUI application 201 once, and the number of development steps can be reduced compared to the conventional technology.

That is, since there is only one type of GUI application (screen drawing processing unit 220, etc.) on the terminal 120, regardless of the type of embedded device 110, changing and retesting the GUI application 201 at the time of OS version upgrade, etc. You only need to do it once, and you can make changes and retest once.

FIG. 18 is a diagram illustrating an example of the GUI application update process of the screen display system according to the embodiment. When updating the GUI application, the changed GUI application 201B (screen drawing processing unit 220, etc.) on the terminal 120 side is uploaded from the external computer 1710, etc. to the application download server 1801. Then, the terminal 120 can download the GUI application 201B (screen drawing processing unit 220, etc.) from the application download server 1801 and install it on the terminal 120.

Next, FIGS. 19 to 21 are explanatory diagrams of a comparison of development man-hours for different types of embedded devices. FIG. 19 shows a system configuration of different types of embedded devices according to the prior art, and FIG. 20 shows a system configuration of different types of embedded devices according to an embodiment. FIG. 21 shows a chart showing a comparison of development man-hours between the conventional technology and the embodiment.

Conventionally, as shown in FIG. 19, three embedded devices 110 (110A to 110C), for example, embedded device 110A includes one sensor A (111) and one LED A (112), The embedded device 110B includes two sensors B1 and B2 (111) and one LED B (112), and the embedded device 110C includes one sensor C (111) and two LEDs C1 and C2 ( 112), and performs different controls for different types.

The plurality of embedded devices 110 (110A to 110C) of different types are individually communicatively connected to an external computer 1710 such as a PC. Each external computer 1710 has three GUI applications 1521A-1521C corresponding to embedded devices 110A-110C, respectively.

Also in the system configuration of the embodiment shown in FIG. 20, the three embedded devices 110 have three different GUI applications 201A to 201C as in the prior art (FIG. 19), but the GUI application on the terminal 120 side is This is a common GUI application (screen drawing processing unit 220, etc.).

Using FIG. 21, the development man-hours for the system configurations of the conventional technology (FIG. 19) and the embodiment (FIG. 20) will be compared. Comparing the conventional technology and the embodiment, the development man-hours for the embedded devices 110A to 110C are the same for all three types.

Then, on the external computer 1710 side of the conventional technology, it is necessary to develop three types of GUI applications 1721A to 1721C that are different for each external computer 1710. Here, in the conventional technology, each of the GUI applications 1721A to 1721C requires development of component placement and screen drawing processing.

On the other hand, on the terminal 120 side of the embodiment, in response to the fact that the three embedded devices 110 have different arrangement configurations of the sensors 111 and LEDs 112, component placement processing k1 corresponding to each of the three embedded devices 110 is performed. .about.k3 development man-hours are required, but the screen drawing processing of the screen drawing processing section 220 requires a common development man-hour.

In this way, in the embodiment, the screen drawing process for the GUI application on the terminal 120 side can be made common to different types of embedded devices 110, and development can be completed in one go. Furthermore, by separating the component placement processing and the screen drawing processing, the parts affected by different system configurations can be minimized, and the number of development steps can be reduced compared to the conventional technology.

As shown in FIG. 21, when comparing the conventional technology and the embodiment, the difference in man-hours increases in proportion to the number of types of embedded devices 110. That is, according to the embodiment, the more types of embedded devices 110 there are, the more the development man-hour can be reduced compared to the conventional technology.

According to the embodiment described above, in the initial state, the device shares initial display list information including character strings and images regarding GUI screen construction and arrangement of each component with the terminal. After operation, the device selects screen information including character strings and images corresponding to a predetermined state from the initial display list information and transmits the selected screen information to the terminal. Based on the received screen information, the terminal refers to the shared initial display list information and displays the operating status of the device on the screen. In this way, by sharing the initial display list information between the device and the terminal in advance, the terminal can display the status of the device, and the amount of data related to screen information to be displayed can be reduced.

Additionally, the device does not display the GUI screen, but only displays the GUI screen on the terminal. For example, the device is an embedded device that has limited uses and does not have a display. Furthermore, the device can be controlled in accordance with operations on the GUI screen of the terminal. Thereby, there is no need to perform drawing processing or the like for displaying a GUI screen on the device side, and the processing load on the device can be reduced, and the cost of the device can be reduced. Devices that perform predetermined control can easily perform control operations by displaying control details on a GUI screen, but in the embodiment, the device only performs processing to generate a GUI screen, No drawing processing is performed. In addition, the device was configured without a display section. This allows the hardware configuration of the device to be simplified and costs to be reduced.

That is, in the embodiment, in a system configuration in which a terminal is communicatively connected to a device, GUI screen generation processing is placed on the device side, and GUI screen drawing processing is placed on the terminal side, so that the GUI function can be transferred to the device. The cost of the entire system was reduced by efficiently separating and arranging the terminals. Note that the terminal can perform drawing processing using a general-purpose smart device, so the cost does not increase.

Also, when updating the GUI screen, the device generates only the updated object, shape, text characters, coordinates, color, etc., and sends it to the terminal. This makes it possible to transfer data from the device to the terminal with a smaller amount of data, which further reduces the amount of data when updating the GUI screen.

Furthermore, the device assigns an identifier to each of the character strings and images included in the initial display list information, and sends the identifiers to the terminal as screen information.The terminal then displays the shared initial display list based on the identifier of the received screen information. Refers to the information and performs screen display processing corresponding to the device status. As a result, when updating screen information due to a change in device status, it is only necessary to send an identifier, further reducing the amount of data to be transferred.

Additionally, a single terminal can be connected to devices from multiple different systems (types); for example, the terminal can switch communications, display a GUI screen generated by a single connected device, and control the device. . Furthermore, it is possible to display and control GUI screens of different devices using the same terminal by switching communication, and it becomes possible to reduce the cost of the entire system including a plurality of devices.

Further, according to the embodiment, in a GUI application for displaying a GUI screen, the configuration related to screen construction is mainly provided on the device side, and the configuration related to screen drawing is provided on the terminal side, so that the GUI application functions. are arranged separately. As a result, even if the number of types increases as the device side responds to various systems, the terminal side can handle this with a single-function GUI application and perform drawing processing.

As a result, even if it becomes necessary to change the GUI application due to an OS version update, for example, the GUI application on the terminal side is of one type regardless of the type of device, so it can be changed only once. All you need to do is change and retest. In this regard, in the conventional technology, it is necessary to change and retest all the GUI applications for each type of device, and as the types of devices increase, the number of development steps increases.

Furthermore, according to the embodiment, the GUI screen generated by the device can be displayed on the terminal, and the user can operate the device while viewing the display screen of the terminal. At this time, the GUI screen sent from the device to the terminal consists of shapes, text characters, coordinates, colors, etc. for each object, and the amount of data is smaller than image data. This makes it possible to transfer GUI screen data from the device to the terminal with a small amount of data. Furthermore, since the amount of data transferred is small, even if the transfer speed between the device and the terminal is slow, the data can be transferred in a short time.

A device and a terminal that are connected for communication can control the operation of the device by operating the terminal. As a result, even if the user is located far from the location where the device is installed, the user can monitor the operating status of the device by looking at the display screen of the terminal, and can also control the operation of the device.

Additionally, since the amount of GUI screen data transferred from the device to the terminal can be reduced, the user can control the device by operating the terminal at a remote location without having to go to the location where the device is installed. It will also be possible to perform remote maintenance from a remote location using a terminal in the event of a malfunction of the equipment. Furthermore, since the GUI screen of the device can be displayed on the terminal, it is possible to reduce costs without providing a display screen for the device. Furthermore, by connecting a terminal to multiple devices, it becomes possible to monitor multiple devices with one terminal.

Note that the screen display program described in this embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer, workstation, or PC (including a tablet and a smartphone). In addition, this screen display method can be used to display data recorded on a computer-readable recording medium such as a hard disk, flexible disk, CD-ROM (Compact Disc-Read Only Memory), or DVD (Digital Versatile Disk), and read from the recording medium by the computer. It is executed by

As described above, the present invention is useful when applied to a technique for transmitting a display screen generated by a device that performs predetermined control to a terminal.

100 Screen display system 110 Equipment (embedded equipment)
111 Sensor 112 LED
120 Terminal 121 Display unit 201 GUI application 211 Screen construction processing unit 220 Screen drawing processing unit 221 Operation unit 301, 311 CPU
302,312 ROM
303,313 RAM
304,314 Storage unit 305,315 Communication interface 308,318 Bus 310 Network 316 Touch panel 317 Display 500 List of character strings and images (Initial display list information K0)
510 Character string list 520 Image list K GUI screen information k Parts placement information S Operation information

Claims (9)

1. In screen display systems that transfer screen information from devices to terminals,
   The equipment includes:
   a control unit that generates screen information indicating the control status of the own machine;
   a communication unit that transmits screen information generated by the control unit to the terminal,
   The terminal is
   a processing unit that generates a display screen based on the screen information received from the device;
   a display unit that displays the display screen generated by the processing unit;
   The control unit of the device includes:
   In an initial state, initial display list information including character strings and images regarding GUI (Graphic User Interface) screen construction and arrangement of each component is shared with the terminal;
   After operation of the device, selecting screen information including the character string and the image corresponding to a predetermined state from the initial display list information and transmitting it to the terminal;
   The processing unit of the terminal includes:
   A screen display system characterized in that, based on the received screen information, the operating state of the device is displayed on the screen by referring to the shared initial display list information.
2. The device does not have a display section for the screen information within its own device,
   2. The terminal according to claim 1, wherein the terminal refers to the shared initial display list information based on the received screen information, performs screen display processing corresponding to the state of the device, and displays the screen on the display unit. Screen display system described.
3. The device assigns an identifier to each character string and image included in the initial display list information, and transmits the identifier to the terminal as the screen information,
   The screen according to claim 1, wherein the terminal refers to the shared initial display list information based on the identifier of the received screen information and performs screen display processing corresponding to the state of the device. display system.
4. The screen display system according to claim 1, wherein the device transmits information regarding the shape, character string, display coordinates, and color of each component to the terminal as the initial display list information.
5. The device updates the screen information in response to the change in the state, and generates only information regarding the shape, text, display coordinates, and color of the component to be updated as the screen information, and transmits it to the terminal. The screen display system according to claim 1, characterized in that:
6. The terminal has an operation section, and transmits operation information from the operation section to the device,
   2. The device according to claim 1, wherein the device performs control based on the operation information received from the terminal, updates the screen information based on the control, and transmits the updated screen information to the terminal. screen display system.
7. The terminal is communicatively connected to a plurality of the devices, displays display information of the communicatively connected devices on the display section, and controls the devices by operating the operation section. 6. The screen display system described in 6.
8. The screen display system according to claim 1, wherein the device is an embedded device that performs predetermined control with limited use.
9. In the screen display method that transfers screen information from the device to the terminal,
   The equipment includes:
   Generates screen information showing the control status of own machine,
   Send the generated screen information to the terminal,
   The terminal is
   displaying a display screen based on the screen information received from the device;
   The equipment includes:
   In an initial state, initial display list information including character strings and images regarding GUI (Graphic User Interface) screen construction and arrangement of each component is shared with the terminal;
   After operation of the device, selecting screen information including the character string and the image corresponding to a predetermined state from the initial display list information and transmitting it to the terminal;
   The terminal is
   A screen display method characterized in that, based on the received screen information, the operating state of the device is displayed on the screen by referring to the shared initial display list information.

Images