JP2023135142A - Information processing apparatus, image processing apparatus, control method, system, and program - Google Patents

Abstract

To display a screen in which a user's visibility is not impaired even when screen transition occurs with a screen enlarged and displayed in one device of two devices connected for remote control.SOLUTION: A program causes a computer of a terminal performing communication with an information processing apparatus to execute a control method, and the control method includes: a receiving step of receiving, from the information processing apparatus, screen information corresponding to a screen displayed on a display of the information processing apparatus; a display step of displaying the screen based on the screen information received in the receiving step; and a changing step of changing the enlargement ratio of the screen displayed in the display step. When the screen displayed on the display of the information processing apparatus is transitioned and the screen information on the screen is received in the receiving step, the enlargement ratio of the screen to be displayed is reset before displaying the screen in the display step.SELECTED DRAWING: Figure 8B

Description

The present disclosure relates to a technique for displaying an operation screen of an image processing apparatus to a remote client.

2. Description of the Related Art A method of remotely controlling an image forming apparatus or the like using a PC, a mobile terminal, or the like is known. This method is realized, for example, by VNC (Virtual Network Computing) using the RFB (Remote FrameBuffer) protocol. Specifically, when a mobile terminal with a VNC client function connects to an image forming apparatus with a VNC server function via VNC, screen information of the operation screen displayed on the operation panel of the image forming apparatus connected via VNC is transmitted from the image forming apparatus. Receive. The mobile terminal then displays the same screen as the operation screen as a remote operation screen based on the received screen information. For example, when a user presses an OK button on a remote control screen of a mobile terminal, the remote control screen is enlarged by pinching out.

Regarding display technology on mobile terminals, Patent Document 1 discloses whether the operation screen of an image forming apparatus is displayed in an enlarged size or in a standard size based on the error frequency of press operations performed by the user over a certain period of time in the past. Discloses technology to switch between.

Japanese Patent Application Publication No. 2013-114554

When a predetermined user operation is performed on the remote control screen of the mobile terminal, the mobile terminal transmits input information corresponding to the predetermined user operation to the image forming apparatus. Then, the mobile terminal receives from the image forming apparatus the screen information of the operation screen after the image forming apparatus executes the process corresponding to the input information and the screen transitions, and displays the screen corresponding to the screen information of the received operation screen. Transition. However, the display unit of the mobile terminal displays the screen after the screen transition in the same screen size as before the screen transition. Therefore, if the remote control screen is enlarged and displayed before the screen transition, the user may not be able to grasp the overall image of the screen after the screen transition, or may not notice that the screen has changed in the first place.

A program according to an aspect of the present disclosure is a program for causing a computer of a terminal that communicates with an information processing device to execute a control method, the control method being performed on a screen displayed on a display unit of the information processing device. a receiving step of receiving screen information corresponding to from the information processing device, a displaying step of displaying a screen based on the screen information received in the receiving step, and a change of changing the magnification rate of the screen displayed in the displaying step. step, when the screen displayed on the display unit of the information processing device is transitioned and the screen information of the screen is received in the receiving step, the display step resets the magnification rate of the screen to be displayed. It is characterized by displaying a screen.

According to the present disclosure, even if a screen transition occurs when one of two devices connected for remote control is enlarged, a screen that does not impair the user's visibility can be displayed. .

FIG. 1 is a diagram showing an example of a schematic configuration of a system. FIG. 2 is a block diagram showing an example of the hardware configuration of a mobile terminal. FIG. 2 is a block diagram showing an example of a software configuration of a mobile terminal. FIG. 2 is a block diagram showing an example of a hardware configuration of an MFP. FIG. 2 is a block diagram showing an example of a software configuration of an MFP. FIG. 3 is a diagram showing an example of a screen displayed by a mobile terminal. FIG. 3 is a diagram showing an example of a screen displayed by a mobile terminal. 3 is a flowchart showing the flow of processing executed by a mobile terminal. 3 is a flowchart showing the flow of processing executed by a mobile terminal. 3 is a flowchart showing the flow of processing executed by the MFP. 3 is a flowchart showing the flow of processing executed by a mobile terminal. It is a flowchart which shows the flow of processing performed by a mobile terminal. It is a flowchart which shows the flow of processing performed by a mobile terminal. 3 is a flowchart showing the flow of processing executed by a mobile terminal. 3 is a flowchart showing the flow of processing executed by the MFP. FIG. 2 is a diagram illustrating the relationship between an example screen displayed by a mobile terminal and an example screen displayed by an MFP.

Hereinafter, embodiments for carrying out the present disclosure will be described using the drawings. However, the components described in this embodiment are merely examples, and the scope of this disclosure is not intended to be limited thereto. In the following description, an MFP (multi-function peripheral) will be used as an example of an image processing apparatus, but the present invention is not limited to this. It may be an image forming apparatus that has an operation panel that displays an operation screen and that can be connected to a mobile terminal via VNC.

(Embodiment 1)
An example of the configuration of a communication system according to this embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a schematic configuration of a communication system according to this embodiment. In this communication system, a mobile terminal 100 and an MFP (multifunction peripheral) 110, which is an image processing device, are connected to an access point 120, and the mobile terminal 100 and MFP 110 communicate via the access point 120. However, the mobile terminal 100 is connected to the access point 120 via wireless communication. The MFP 110 is connected to an access point 120 via a LAN (Local Area Network). Note that the connection method (form) between MFP 110 and access point 120 is not limited to wired connection, and may be wireless. In that case, MFP 110 includes a wireless communication device for connecting to access point 120 via wireless communication.

The mobile terminal 100 has a client function for remote control. For example, VNC client software compatible with the RFB protocol is installed on the mobile terminal 100. When the mobile terminal 100 executes the VNC client software and establishes a VNC connection provided by the VNC client software to the MFP 110, the mobile terminal 100 displays the same operation screen as the operation screen displayed on the operation panel of the MFP 110 as a remote operation screen. MFP 110 has a VNC server function. When connected via VNC from the mobile terminal 100, the MFP 110 transmits screen information of the operation screen displayed on the operation panel to the client mobile terminal 100. Packets of the RFB protocol are packaged with the HTTP protocol.

<Hardware configuration of mobile terminal>
FIG. 2 is a diagram showing an example of the hardware configuration of the mobile terminal 100. Note that, although the mobile terminal 100 of this embodiment is assumed to be a device such as a smartphone or a tablet PC, the present invention is not limited thereto. Mobile terminal 100 may be any form of information processing device as long as it has a function of communicating with MFP 110.

The mobile terminal 100 includes a CPU 201, a ROM 202, a RAM 203, an HDD 204, an RTC 205, an operation panel 206, a speaker 207, a microphone 208, a camera 209, and a wireless communication section 211.

The CPU 201 is a central processing unit, reads out a control program stored in the ROM 202, and executes various processes for controlling the operation of the mobile terminal 100. A ROM (Read Only Memory) 202 stores a control program. A RAM (Random Access Memory) 203 is used as a temporary storage area such as a main memory or a work area of the CPU 201. A HDD (Hard Disk Drive) 204 stores various data such as photographs and electronic documents. The HDD 204 also stores an OS (Operating System) 350 and an image processing device application 300, which will be described later. An RTC (Real Time Clock) 205 measures time.

In the case of the mobile terminal 100, it is assumed that one CPU 201 executes each process shown in a flowchart described later, but other modes may be used. For example, a plurality of CPUs may cooperate to execute each process shown in a flowchart described later.

The operation panel 206 is a user interface that has a touch panel function that can detect a user's touch operation, and displays various screens provided by the OS (Operating System) 350 and the image processing device application 300. Since the operation panel 206 displays a screen, it can also be said to have the function of a display section. By inputting a touch operation to the operation panel 206, the user can input a desired operation instruction to the mobile terminal 100. Note that the mobile terminal 100 may include a hardware key (not shown), and the user may input operation instructions to the mobile terminal 100 using this hardware key.

The speaker 207 and microphone 208 are used when the user makes a call to another mobile terminal or landline phone. The camera 209 captures an image according to a user's instruction to capture an image. A photograph taken by the camera 209 is stored in a predetermined area of the HDD 204. The wireless communication unit 211 performs wireless communication such as a wireless communication LAN.

<Software configuration of mobile terminal>
An example of the software configuration of the mobile terminal 100 will be described with reference to the drawings. FIG. 3 is a functional block diagram of software realized by the CPU 201 reading out a control program stored in the ROM 202 or HDD 204.

The OS 350 is software for controlling the overall operation of the mobile terminal 100. Various applications, including the image processing device application 300, can be installed on the mobile terminal 100. The OS 350 exchanges information with these applications and changes the screen displayed on the operation panel 206 according to instructions received from the applications. The OS 350 also includes a group of device drivers for controlling various hardware. The OS 350 provides an API (Application Programming Interface) for using various hardware for applications running on the OS. In this embodiment, a wireless LAN control unit 351 is included as a device driver group. The wireless LAN control unit 351 is a device driver for controlling the wireless communication unit 211.

The image processing device application 300 is an application installed on the mobile terminal 100. Operations such as printing, scanning, and remote control of the operation screen of the MFP 110 can be performed on the MFP 110 from the image processing device application 300. Various applications other than the image processing device application 300 are installed on the mobile terminal 100, but the explanation thereof will be omitted.

The software configuration of the image processing device application 300 will be explained in more detail. The image processing device application 300 includes a screen control unit 301, a communication unit 302, a clock unit 304, a print job generation unit 305, an application storage unit 306, a scan job control unit 307, a document management unit 308, a device search control unit 311, and a remote control unit. 313.

The screen control unit 301 controls the screen displayed on the operation panel 206 via the OS 350 and manages the display state. The screen control unit 301 displays an image processing device application screen on the operation panel 206. Further, the screen control unit 301 determines an operation instruction input by the user via the operation panel 206. The communication unit 302 controls wireless communication by the wireless communication unit 211 via the OS 350 to transmit and receive commands.

The device search control unit 311 provides a function of searching for image processing apparatuses on the LAN 10 to which the mobile terminal 100 participates (connects), and searching for image processing apparatuses to which the mobile terminal 100 can connect via VNC. Specifically, the device search control unit 311 generates search data for searching for an image processing device to connect to, and transmits the generated search data to a device on the LAN 10 in which the mobile terminal 100 participates via the wireless communication unit 211. do. Then, the device search control unit 311 searches for an image processing device that can be connected via VNC by receiving a response from the corresponding image processing device. The search data may be broadcast to all devices on the LAN 10, or may be sent by specifying the address of one device.

The print job generation unit 305 generates a print job. The print job generated by the print job generation unit 305 is transmitted to the MFP 110 by the wireless communication unit 211, and the MFP 110 is caused to execute printing. The application storage unit 306 temporarily stores various information. The scan job control unit 307 instructs the MFP 110 to scan via the wireless communication unit 211, and controls scan data received from the MFP 110. When saving scan data, it is temporarily held in the application storage unit 306. The retained scan data is managed by the document management unit 308.

The remote operation unit 313 operates a first screen corresponding to all or part of the UI screen as a remote operation screen based on image data of the operation screen (UI screen) displayed on the operation panel 3070 acquired from the MFP 110. Displayed on panel 206. The remote control unit 313 can also be said to have the function of a display control unit since it causes the operation panel 206 to display the first screen. The remote control unit 313 accepts user operations on the screen displayed on the operation panel 206. The remote control unit 313 can also be said to have the function of a reception unit since it accepts user operations. Additionally, the remote control unit 313 transmits instruction information (input information) of user operations received on the screen displayed on the operation panel 206 to the MFP 110 . The remote control unit 313 remotely controls the operation panel of the MFP 110 by executing these operations. The acquired screen data and the input operation information are transmitted to and received from the MFP 110 by the wireless communication unit 211. When the user operation on the screen displayed on the operation panel 206 is pinch-in/pinch-out, the screen displayed on the operation panel 206 of the mobile terminal 100 is enlarged or reduced. The clock unit 304 clocks time via the RTC 205 .

<MFP hardware configuration>
FIG. 4 is a diagram showing an example of the hardware configuration of the MFP 110. Note that in this embodiment, the controller of the multifunction device also has a similar configuration. The MFP 110 can be said to be an information processing device since it processes information as described in detail later.

As shown in FIG. 4, the MFP 110 includes a system bus 3010, a CPU 3020, a ROM 3030, a RAM 3040, and a network interface card (NIC) 3050. The MFP 110 further includes an external input controller (PANELC) 3060, a touch panel (PANEL) 3070, a display controller (DISPC) 3080, and a display module (DISPLAY) 3090. The MFP 110 includes a disk controller 3100, a large-scale storage device 3110, a hardkey 3120, a printer 3200, a scanner (image reading unit) 3300, and a wireless communication module (WLAN) 3400.

CPU 3020 executes software stored in ROM 3030 or large-scale storage device 3110 such as a hard disk. CPU 3020 generally controls each device connected to system bus 3010.

The RAM 3040 functions as the main memory, work area, etc. of the CPU 3020. External input controller 3060 controls input of instructions from various hard keys 3120 provided in MFP 110 or operation panel 3070 having a function of detecting touch operations. The hard keys 3120 are used, for example, to press the "home" key to display various functions such as copying and scanning, or to display the home screen that is the entrance to the settings menu, the "reset" key to cancel settings and return to the state before the settings, and the "reset" key to cancel settings and return to the state before the settings were entered. Has a "clear" key to cancel letters and numbers. The hard keys 3120 are, for example, a "back" key to return from the currently displayed screen to the previous screen, a "confirm status" key to display information such as printing status and network settings, and a key to enter numbers and characters. It has a numeric keypad, a "stop" key that monitors operations such as printing. Further, the hard keys 3120 include, for example, a "power saving" key for setting a sleep mode (power saving state) and a "start" key to press when reading or copying a document. In addition, the hard keys 3120 include, for example, the "#" key that is pressed when entering symbols such as "@" and "/", the "*" key that switches character input modes (numbers, kana, alphabets), etc. be.

Since the operation panel 3070 displays a user interface screen (UI screen), it can also be said to have the function of a display unit. A display controller 3080 controls the display of a display module 3090 configured with, for example, a liquid crystal display. Disk controller 3100 controls mass storage device 3110. When an authentication key (not shown) on the hard key 3120 is pressed during login, a logout process is performed.

The network interface card 3050 exchanges data bidirectionally with other network devices, file servers, etc. via the LAN (network) 10. By connecting the wireless communication module 3400 to an access point or operating in an access point mode, the image processing apparatus operates as an access point and can directly connect to the mobile terminal 100 for wireless communication. The printer 3200 is a printing unit that prints on paper (recording medium) using an electrophotographic method. The printing method does not particularly have to be an electrophotographic method.

The scanner 3300 is an image reading unit for reading an image printed on paper. In many cases, the scanner 3300 is equipped with an ADF (auto document feeder) (not shown) as an option, and can automatically read multiple sheets of originals.

Note that the large-scale storage device 3110 may also be used as a temporary storage location for images in some cases.

<MFP software configuration>
FIG. 5 is a block diagram showing the software configuration of MFP 110. The CPU 3020 functions as each module in FIG. 5 by executing software stored in the ROM 3030. The MFP 110 includes a network control section 500, a UI processing section 510, a remote operation processing section 520, and a user authentication section 530.

The UI processing unit 510 analyzes operation information (input information) received from an input from the external input controller 3060 or an operation information reception unit 522 described later. If the UI processing unit 510 determines that a screen change is necessary based on the analysis results, it generates screen data (image data) for the UI screen, and displays the screen represented by the generated screen data on the display module (display) 3090. to be displayed. That is, the display module 3090 displays the UI screen after the screen transition.

The remote operation processing section 520 includes each module of a connection processing section 523, an operation information receiving section 522, and a screen transmitting section 521. The remote control processing unit 520 is a VNC server module that performs communication using the RFB protocol, and is a software module that communicates with a VNC client operating on the mobile terminal 100 via a network. When the connection processing unit 523 receives a connection request from a VNC client, it performs connection processing to connect to the VNC client. When the connection processing unit 523 receives a disconnection request from the VNC client, it performs disconnection processing to disconnect the connection with the VNC client. The operation information receiving unit 522 receives the operation information transmitted from the VNC client, and notifies the UI processing unit 510 that the operation information has been received. The screen transmitter 521 reads the screen data generated by the UI processor 510 from the RAM 3040 and transmits it to the VNC client.

The user authentication unit (login processing unit) 530 is a module that performs user authentication and login processing, and confirms user information received from the UI processing unit 510. The user authentication unit 530 executes login processing when the user authentication is successful. Further, when the user authentication unit 530 receives a logout process from the UI processing unit 510, the user authentication unit 530 executes the logout process.

Network control unit 500 is a module for controlling network I/F 3050.

Although not shown in FIG. 5, the MFP 110 also includes a printer module that controls the printer 3200 and a scanner module that controls the scanner 3300.

<About the remote control screen>
The remote control screen displayed on the operation panel 206 of the mobile terminal 100 will be explained using figures. Specifically, the same screen as the operation screen displayed on the operation panel 3070 of the MFP 110 (hereinafter referred to as the VNC screen) is displayed on the operation panel 206 when the mobile terminal 100 establishes a VNC connection with the MFP 110 using the RFB protocol. , will be explained using figures. Since the VNC screen is a screen for remotely controlling the MFP 110, it can also be called a remote control screen. 6 and 7 are diagrams illustrating a remote control screen displayed on the operation panel 206 of the mobile terminal 100 according to the present embodiment. FIG. 13 is a diagram before applying this embodiment, and is a diagram illustrating the relationship between an example of a screen displayed by a mobile terminal and an example of a screen displayed by an MFP. FIG. 13(a) shows an example of the screen of the mobile terminal before the screen transition, FIG. 13(b) shows an example of the MFP screen after the screen transition, and FIG. 13(c) shows the screen of the mobile terminal after the screen transition. Give an example. FIG. 13A shows a scene in which the user performs an operation on the OK button. In FIG. 13(b), a two-dot chain line indicates an area corresponding to the operation panel of the mobile terminal. In addition, as shown in FIGS. 13(b) and 13(c), the location where the user wants to operate is displayed on the operation panel of the MFP but not on the operation panel of the mobile terminal. It shows that there is.

The destination confirmation screen (full screen) 650 is displayed on the operation panel 206 of the mobile terminal 100, and is the same screen as the mail destination confirmation screen (not shown) displayed on the operation panel of the MFP. When the display area of the OK button 601 is pressed by the user on the destination confirmation screen (full screen) 650, the scan and send screen (full screen) 630 that transitioned from the destination confirmation screen (full screen) 650 is displayed on the mobile terminal 100. is displayed on the operation panel 206 of.

If the operation screen displayed on the operation panel 3070 is updated while the same operation screen as the operation screen displayed on the operation panel 3070 of the MFP 110 is displayed on the operation panel 206 of the mobile terminal 100, the operation screen displayed on the operation panel 3070 of the mobile terminal 100 is updated. executes the following processing. That is, mobile terminal 100 receives the updated portion of image data from MFP 110, and updates the operation screen displayed on operation panel 206 based on the received updated image data. Therefore, as shown in FIG. 13, even if the operation screen displayed on the operation panel 3070 of the MFP 110 changes, the operation panel of the conventional mobile terminal displays the screen while maintaining the enlargement ratio. As a result, users of mobile terminals may be unable to find the location they want to operate from within the screen displayed on the operation panel of the mobile terminal, or may not be able to determine whether the screen on the operation panel of the mobile terminal is the screen after the transition. There were times when it was impossible to make a decision.

The destination confirmation screen (enlarged display) 600 is a screen in which the destination confirmation screen (full screen) 650 is enlarged by performing a pinch-out operation near the OK button 601 on the destination confirmation screen (full screen) 650 . That is, the destination confirmation screen (enlarged display) 600 is a state in which a part of the first screen corresponding to all or part of the UI screen of the MFP is displayed in an enlarged manner. When the area where the OK button 601 is displayed on the destination confirmation screen (enlarged display) 600 is pressed, a scan and send screen (enlarged display) 610 is displayed with the same magnification rate and enlargement range as the destination confirmation screen (enlarged display) 600. is displayed. At this time, if the image data sent from the MFP is full screen size, it is assumed that the operation screen of the MFP 110 has changed, the magnification rate of the VNC screen is reset, and the scan and send screen (full screen) 630 is displayed. That is, when screen data for the entire screen is sent from the MFP 110 instead of screen data for only the updated area, it is determined that a screen transition has occurred, and the screen after the screen transition is displayed. Further, the destination confirmation screen (enlarged display, soft key display) 620 is a screen in which soft keys (hereinafter referred to as soft keys) imitating the hard keys 3120 of the MFP 110 are superimposed and displayed on the destination confirmation screen (enlarged display) 600. be. Switching between displaying and non-displaying the soft keys is performed by pressing the open/close button 602. Since the operation screen of the MFP 110 may change when any of the soft keys is pressed, the magnification ratio of the VNC screen is reset and the scan and send screen (full screen, soft key display) 640 is displayed. shall be carried out. For example, the "home" soft key is a button for displaying a home screen on the operation panel of the MFP. Therefore, when the "home" soft key is pressed, a screen transition occurs, so the mobile terminal 100 resets the enlargement ratio of the VNC screen. Furthermore, when the user is performing input using the "numeric keypad" soft keys, it is better to be able to grasp the entire image of the screen, so the mobile terminal 100 may similarly reset the magnification ratio of the VNC screen. It is also possible to set whether or not to reset the enlargement ratio of the VNC screen for each soft key input.

A setting may be provided to enable or disable a function of resetting the enlargement ratio of the VNC screen when a screen transition occurs. Similarly, a setting may be provided to enable or disable the function of resetting the enlargement ratio of the VNC screen by inputting a soft key.

<Remote control processing using a mobile terminal>
Processing performed when the mobile terminal 100 remotely controls the MFP 110 according to the present embodiment will be described with reference to the drawings. Specifically, a process will be described in which the mobile terminal 100 establishes a VNC connection with the MFP 110 using the RFB protocol, resets the screen display and VNC magnification rate, and ends the VNC connection. 8A and 8B are flowcharts showing the flow of processing executed by the mobile terminal 100 according to the present embodiment. Each operation (step) shown in the flowcharts of FIGS. 8A and 8B is realized by the CPU 201 reading a program for realizing each control module stored in the ROM 202 or HDD 204 into the RAM 203 and executing it. Furthermore, some or all of the functions of the steps in FIGS. 8A and 8B may be realized by hardware such as an ASIC or an electronic circuit. Note that the symbol "S" in the description of each process means a step in the flowchart, and the same applies to subsequent flowcharts.

In S801, the CPU 201 starts the image processing device application 300 based on an instruction from the user. In S802, the CPU 201 searches for a device to be remotely controlled. In S803, the CPU 201 determines whether the user has selected a device to be remotely controlled. If a determination result is obtained that a device to be remotely controlled has been selected (YES in S803), the process moves to S804. If the determination result is that the device to be remotely controlled is not selected (NO in S803), the process returns to S803. That is, the process of S803 is repeatedly executed until the user selects a device to be remotely controlled.

In S804, the CPU 201 determines whether a remote operation menu (not shown) displayed on the operation panel 206 of the mobile terminal 100 has been selected by a user operation. If it is determined that the remote operation menu has been selected (YES in S804), the process moves to S805. If the determination result is that the remote operation menu is not selected (NO in S804), the process returns to S804. That is, the process of S804 is repeatedly executed until the remote operation menu is selected by user operation.

In S805, the CPU 201 issues an RFB connection request to the device selected in S803 based on the RFB protocol.

In S806, the CPU 201 determines whether the connection request in S805 has been accepted by the device selected in S803. If the determination result is that the connection request has been accepted (YES in S806), the process moves to S807. If the determination result is that the connection request has not been accepted (NO in S806), the process moves to S828. Whether or not to accept the connection request is determined based on the response data of the connection request. For example, if the response data of the connection request indicates that the protocol versions do not match, it is determined that the connection request has not been accepted. If the response data of the connection request indicates that the protocol versions match, it is determined that the connection request has been accepted. In S828, the CPU 201 notifies the user of the mobile terminal 100 of the connection refusal error. Specifically, the CPU 201 displays an error message indicating a connection refusal error on the operation panel 206 of the mobile terminal 100. When the error display is executed, the CPU 201 ends the processing shown in FIGS. 8A and 8B.

In S807, the CPU 201 performs initialization processing in the RFB protocol. In the initialization process, client information is transmitted and server information is received. In this embodiment, it is assumed that the client information is information on the mobile terminal 100 and the server information is information on the MFP 110.

In S808, the CPU 201 receives screen data representing the operation screen displayed on the operation panel 3070 of the MFP 110, transmitted from the MFP 110 (server).

In S809, the CPU 201 determines whether the image data received in S808 is the first data received after starting the image processing apparatus application. If it is determined that the image data is the first image data (YES in S809), the process moves to S810. If it is determined that the image data is not the first image data (NO in S809), the process moves to S811.

In S810, the CPU 201 stores the screen size of the image data transmitted from the MFP 110 (server) in the application storage unit 306 as the full screen size.

In S811, the CPU 201 determines whether the screen size of the image data transmitted from the MFP 110 (server) matches the size of the entire screen saved in S810. If it is determined that they match (YES in S811), the process moves to S812. If it is determined that they do not match (NO in S811), the process proceeds to S813 while maintaining the enlargement rate of the VNC screen. That is, here, if the screen displayed on the MFP 110 changes and the mobile terminal 100 receives screen data corresponding to the screen after the transition, the process advances to S812. On the other hand, if a button included in a screen displayed on the MFP 110 is selected and only the button is highlighted without the screen transitioning, and the mobile terminal 100 receives screen data corresponding to the screen on which the button is highlighted, the process advances to S813. . Note that the screen data corresponding to the screen on which the button is displayed in reverse is screen data that is the difference between before and after the button is selected. Here, the configuration is such that whether or not the screen displayed on the MFP 110 has changed is determined based on whether or not screen data of the size of the entire screen has been received, but the present invention is not limited to this. For example, when the screen displayed on MFP 110 changes, information indicating that the screen has changed is sent from MFP 110 to mobile terminal 100, and it is determined whether the screen displayed on MFP 110 has changed based on the information. You may also do this.

In S812, the CPU 201 resets the enlargement rate of the VNC screen. For example, if it is determined in S811 that the image data is the full screen size, the enlargement rate of the VNC screen will be reset.

In S813, the CPU 201 updates the VNC screen using the image data received in S808. That is, the CPU 201 displays the screen represented by the received image data as a VNC screen (remote control screen) on the operation panel 206 of the mobile terminal 100.

In S814, the CPU 201 determines whether an input operation by the user has been accepted on the operation panel 206. If a determination result indicating that the input operation has been accepted is obtained (YES in S814), the process moves to S815. If the determination result is that the input operation is not accepted (NO in S814), the process returns to S808.

In S815, the CPU 201 determines whether the input operation received in S814 is a pinch-in or a pinch-out. If the determination result is that it is pinch-in or pinch-out (YES in S815), the process moves to S816. If the determination result is that the pinch is not pinch-in or pinch-out (NO in S815), the process moves to S819.

In S816, the CPU 201 calculates the magnification rate from the pinch-in or pinch-out operation amount. In S817, the CPU 201 calculates the display area of the VNC screen from the pinch-in or pinch-out center and the enlargement rate. In S818, the VNC screen of the display area calculated in S817 is drawn on the operation panel 206. Note that upon completion of the process in S818, the process returns to S814.

In S819, the CPU 201 determines whether the input operation received in S814 is a drag. If it is determined that the input operation is a drag (YES in S819), the process moves to S820. If it is determined that the input operation is not a drag (NO in S819), the process moves to S823. In S820, the CPU 201 determines whether the VNC screen is displayed in an enlarged manner. If it is determined that the VNC screen is displayed in an enlarged manner (YES in S820), the process moves to S821. If it is determined that the VNC screen is not displayed in an enlarged manner (NO in S820), the process moves to S823. In S821, the CPU 201 calculates the display area of the VNC screen from the current enlargement ratio and the amount of drag movement. In S822, the CPU 201 draws the VNC screen of the display area calculated in S821 on the operation panel 206. Note that upon completion of the process in S822, the process returns to S814.

In S823, the CPU 201 determines whether the input operation received in S814 is a press of a soft key imitating a hard key of the MFP 110. That is, it is determined whether a user operation that causes a screen transition has been accepted. If it is determined that a soft key has been pressed (YES in S823), the process moves to S824. If it is determined that the soft key has not been pressed (NO in S823), the process moves to S825.

In S824, the CPU 201 resets the enlargement rate of the VNC screen. For example, if it is determined in S823 that a soft key imitating a hard key of the MFP 110 has been pressed, the magnification rate of the VNC screen will be reset.

In S825, the CPU 201 transmits input information indicating the content of the input operation received in S814 to the MFP 110. For example, when the process reaches S825 after passing through the process of S824, the area information indicating the display area of the VNC screen (first screen) whose enlargement ratio of the VNC screen has been reset and the input information are transmitted in association with each other. It's okay. If it is determined in S811 that the image data received in S809 is full screen size and the process reaches S825 after passing through the processing in S812, area information indicating the display area of the VNC screen in which the enlargement rate of the VNC screen has been reset. The information may be transmitted in association with the input information. In this case, if neither of the processes in S812 or S824 described above has been performed, the input information is associated with area information indicating the display area of the VNC screen (first screen) for which the enlargement rate of the VNC screen has not been reset. and then send it.

In S826, the CPU 201 determines whether the connection has been terminated. Termination of the connection includes, for example, a user operation on a button for terminating the VNC connection on a screen (not shown) displayed on the operation panel 206 of the mobile terminal 100, forced termination of the image processing device application, and the like. If the determination result is that the connection has been terminated (YES in S826), the process moves to S827. If the determination result is that the connection has not been terminated (NO in S826), the process returns to S808.

In S827, the CPU 201 performs VNC disconnection processing and ends the VNC connection. When the process of S827 is completed, the flow shown in FIGS. 8A and 8B ends.

In the flow shown in FIG. 8, when operation information that involves a screen transition of the UI screen is received with a portion of the first screen enlarged, the following information is sent to the MFP 110. . That is, area information indicating the display area of the first screen in which the enlargement instruction for the first screen has been reset is transmitted in association with the operation information. Furthermore, if the operation information is transmitted in association with area information indicating the display area of the first screen where the enlargement instruction for the first screen has been reset, the following image data will be received. . That is, the image data of the UI screen after the screen transition executed by the MFP 110 and displayed on the MFP 110 based on the operation information is received. Further, when image data is received by associating and transmitting area information indicating the display area of the first screen where the enlargement instruction for the first screen has been reset and the operation information, the following: The first screen shown in will be displayed. That is, based on the received image data, the first screen corresponding to the UI screen after the screen transition in which the enlargement instruction is not reflected is displayed on the display unit.

<Processing by MFP>
Processing executed by the MFP 110 according to the present embodiment will be described using figures. FIG. 9 is a flowchart showing the flow of processing executed by the MFP 110 according to this embodiment. Each step shown in the flowchart of FIG. 9 is realized by the CPU 3020 loading a control program stored in a memory such as a ROM 3030 into the RAM 3040 and executing it.

In S901, the CPU 3020 determines whether there is a connection request from the mobile terminal 100 (VNC client). If a determination result indicating that a connection request has been made is obtained (YES in S901), the process moves to S902. If the determination result is that there is no connection request (NO in S901), the process returns to S901. That is, the process of S901 is repeatedly executed until a determination result indicating that a connection request has been made is obtained.

In S902, the CPU 3020 performs negotiation processing and establishes a connection with the mobile terminal 100 (VNC client). In S903, the CPU 3020 performs initialization processing in the RFB protocol. In the initialization process, client information is received and server information is transmitted.

In S904, the CPU 3020 transmits screen data of the screen (operation screen) displayed on the display module 3090 to the mobile terminal 100 (VNC client). For example, when S904 is reached after going through S906 and S907, which will be described later, image data corresponding to the area information (enlargement ratio of the VNC screen) indicated by the display area of the VNC screen (first screen) associated with the input information is transmitted. You may also do so. For example, when the operation screen of the operation panel 3070 of the MFP 110 changes by accepting and executing input information associated with area information indicating the display area of the VNC screen whose enlargement rate has been reset, the following image is displayed. Allow data to be sent. That is, screen information of the operation screen (full screen) after the screen transition is transmitted as image data to the VNC-connected mobile terminal 100. Furthermore, when the operation screen of the operation panel 3070 of the MFP 110 changes by receiving and executing input information associated with area information indicating the display area of the VNC screen for which the enlargement rate of the VNC screen has not been reset, the following Image data is sent. That is, image data of the operation screen (enlarged screen) after the screen transition is transmitted to the VNC-connected mobile terminal 100. Note that the screen displayed on the display module 3090 is also referred to as a second screen.

In S905, CPU 3020 determines whether input information from mobile terminal 100 has been received. If a determination result indicating that input information from the mobile terminal 100 has been received (YES in S905), the process moves to S906. If the determination result is that input information from the mobile terminal 100 is not accepted (NO in S905), the process returns to S904.

In S906, the CPU 3020 executes the input information received from the mobile terminal 100. For example, when input information is executed, the operation screen displayed on operation panel 3070 of MFP 110 transitions to another screen.

In S907, the CPU 3020 determines whether there is a VNC disconnection request. If a determination result indicating that there is a disconnection request is obtained (YES in S907), the process moves to S908. If the determination result is that there is no disconnection request (NO in S907), the process returns to S904.

In S908, the CPU 3020 performs VNC disconnection processing and ends the VNC connection. When the process of S908 is completed, the flow shown in FIG. 9 ends.

In the flow shown in FIG. 9, when the mobile terminal 100 receives operation information that involves a screen transition of the UI screen with a portion of the first screen enlarged, the mobile terminal 100 transmits the following information to the mobile terminal 100. It will be received from 100. That is, area information indicating the display area of the first screen in which the enlargement instruction for the first screen has been reset and the operation information are received in association with each other. Further, when the operation information is received in association with the area information indicating the display area of the first screen where the enlargement instruction for the first screen has been reset, the following image data will be transmitted. . That is, image data of the UI screen after the screen transition executed by the MFP 110 and displayed on the MFP 110 based on the operation information is transmitted.

As explained above, in this embodiment, the following processing is executed. That is, when full-screen image data is transmitted from the MFP 110 to the mobile terminal 100, or when a soft key imitating the hard keys of the MFP 110 is pressed on the mobile terminal 100, the enlargement rate of the VNC screen is changed. Reset. As a result, the input information transmitted from the mobile terminal 100 to the MFP 110 can only be associated with area information indicating the display area of the VNC screen whose enlargement rate has been reset. When the MFP 110 executes the received input information and the operation screen of the operation panel 3070 of the MFP 110 transitions, it transmits screen information of the operation screen (full screen) after the transition to the VNC-connected mobile terminal 100. Then, the mobile terminal 100 displays on the operation panel 206 the screen represented by the screen information of the operation screen (full screen) after the screen transition received from the MFP 110. In this way, even if a screen transition occurs when one of the mobile terminal 100 and MFP 110 connected for remote operation is enlarged, a screen that does not impair the user's visibility can be displayed. Operability can be improved.

(Embodiment 2)
In the first embodiment, the case where the enlargement ratio of the VNC screen is reset when full-screen image data is transmitted from the MFP 110 to the mobile terminal 100 or when a soft key is pressed on the mobile terminal 100 is described. Indicated. However, when the operation screen displayed on the operation panel 3070 of the MFP 110 changes but a portion of the operation screen remains unchanged, only the screen data of the changed portion is transmitted from the MFP 110 to the mobile terminal 100. Therefore, image data of the full screen size is not necessarily transmitted. Further, the soft keys include numeric keys for inputting numbers, and if the magnification rate of the VNC screen is reset every time a number is input, it will not be possible to input a number while displaying an enlarged display.

Therefore, in this embodiment, although the enlargement ratio is reset even when image data of a specific area is received, a mode will be described in which the enlargement ratio of the VNC screen is not reset depending on the type of soft key. Furthermore, the timing at which the enlargement ratio of the VNC screen is reset may be different from the timing expected by the user. Therefore, in this embodiment, a case will also be described in which the user can turn ON/OFF the function of resetting the enlargement ratio of the VNC screen. The basic configuration of this embodiment is the same as that of Embodiment 1, so the explanation will focus on the differences. Note that the user can set in advance on the settings screen whether to turn on the function that resets the enlargement ratio of the VNC screen to enable it, or turn it off to disable the function. shall be. Further, it is assumed that the setting of the reset function of the enlargement ratio of the VNC screen is held in the image processing device application 300.

<Remote control processing using a mobile terminal>
Processing performed when the mobile terminal 100 remotely controls the MFP 110 according to the present embodiment will be described with reference to the drawings. Specifically, a process will be described in which the mobile terminal 100 establishes a VNC connection with the MFP 110 using the RFB protocol, resets the screen display and VNC magnification rate, and ends the VNC connection. 10A and 10B are flowcharts showing the flow of processing executed by the mobile terminal 100 according to this embodiment. Each operation (step) shown in the flowcharts of FIGS. 10A and 10B is realized by the CPU 201 reading a program for realizing each control module stored in the ROM 202 or HDD 204 into the RAM 203 and executing it. Further, some or all of the functions of the steps in FIGS. 10A and 10B may be realized by hardware such as an ASIC or an electronic circuit. Furthermore, since the basic configuration of this flowchart is the same as the flowcharts of FIGS. 8A and 8B, the differences will be mainly explained. That is, S1001 to S1010 are the same as S801 to S810, S1015 to S1024 are the same as S813 to S822, and S1029 to S1031 are the same as S826 to S828, and their explanations will be omitted.

In S1009, the CPU 201 determines whether the image data received in S1008 is the first data received after starting the image processing device application, and if the CPU 201 determines that it is not the first image data (in S1009) NO), the process moves to S1011. Note that if the determination result is that the image data is the first image data (YES in S1009), the process moves to S1010.

In S1011, the CPU 201 determines whether a function for resetting the magnification rate of the VNC screen (hereinafter also referred to as a magnification rate reset function) is set to ON. If it is determined that the enlargement rate reset function is set to ON (YES in S1011), the process moves to S1012. If it is determined that the magnification reset function is set to OFF (NO in S1011), the process moves to S1015.

In S1012, the CPU 201 determines whether the screen size represented by the image data transmitted from the MFP 110 (server) to the mobile terminal 100 is the full screen size. If the determination result is that the screen size of the image data is not the full screen size (NO in S1012), the process moves to S1013. If a determination result is obtained that the screen size of the image data is the full screen size (YES in S1012), the process moves to S1014.

In S1013, the CPU 201 determines whether the image data received in S1008 is image data of a specific area. If it is determined that the image data is of a specific area (S1013: YES), the process moves to S1014. If it is determined that the image data is not of a specific area (NO in S1013), the process moves to S1015. Note that to determine whether the image data is of a specific area, first, information indicating the type of MFP 110 connected to the mobile terminal 100 selected in S1003 is acquired. Next, the type of operation screen of the MFP 110 is determined from the acquired type of MFP, and a map of screen parts is maintained. The process is then performed by comparing whether the image data transmitted from the MFP 110 matches a specific part on the map.

In S1014, the CPU 201 resets the enlargement rate of the VNC screen. For example, if it is determined in S1011 that the VNC screen magnification reset function is set to ON, and it is determined in S1012 that the screen size of the image data is the full screen size, the VNC screen magnification rate is reset. will be done. However, even if it is determined in S1012 that the screen size of the image data is not the full screen size, if it is determined in S1013 that it is image data of a specific area, the enlargement rate of the VNC screen is reset.

In S1022, the CPU 201 determines whether the VNC screen is displayed in an enlarged manner, and if it is determined that the VNC screen is not displayed in an enlarged manner (NO in S1022), the process proceeds to S1025. Note that if a determination result is obtained that the VNC screen is displayed in an enlarged manner (YES in S1022), the process moves to S1023.

In S1025, the CPU 201 determines whether the input operation received in S1016 is a press of a key other than numeric, #, *, and Clear among soft keys imitating the hard keys of the MFP. That is, it is determined whether operation information for a soft key other than a specific soft key has been received. If it is determined that a soft key other than numeric, #, *, or Clear has been pressed (YES in S1025), the process moves to S1026. If it is determined that a soft key other than numeric, #, *, or Clear has been pressed (NO in S1025), the process moves to S1028.

In S1026, the CPU 201 determines whether a function for resetting the magnification rate of the VNC screen (hereinafter also referred to as a magnification rate reset function) is set to ON. If a determination result is obtained that the enlargement rate reset function is set to ON (YES in S1026), the process moves to S1027. If it is determined that the magnification reset function is set to OFF (NO in S1026), the process moves to S1028.

In S1027, the CPU 201 resets the enlargement rate of the VNC screen. For example, in S1025, it is determined that among the soft keys imitating the hard keys of the MFP 110, keys other than numbers, #, *, and Clear have been pressed, and in S1026, it is determined that the magnification reset function is set to ON. The above-mentioned reset is executed when

In S1028, the CPU 201 transmits input information indicating the content of the input operation accepted in S1016 to the MFP 110. For example, if the following two determination results are obtained and the process reaches S1028, input information associated with area information indicating the display area of the VNC screen (first screen) for which the enlargement rate of the VNC screen has been reset will be sent. That is, in the first determination, it is determined that the input operation received in S1016 in S1025 is a press of a key other than numeric, #, *, and Clear among soft keys imitating the hard keys of the MFP. It is to obtain. The second determination is to obtain a determination result that the enlargement rate reset function is set to ON in S1026. In addition, if the following two determination results are obtained and the process reaches S1028, input information associated with area information indicating the display area of the VNC screen (first screen) for which the enlargement rate of the VNC screen has been reset will be sent. That is, the first determination is to obtain a determination result that the enlargement rate reset function is set to ON in S1011. The second determination is to obtain a determination result in S1012 that the image data received in S1008 is the full screen size. In addition, if the following two determination results are obtained and the process reaches S1028 through the process of S1014, area information indicating the display area of the VNC screen (first screen) in which the enlargement rate of the VNC screen has been reset. The input information associated with the will be sent. That is, the first determination is to obtain a determination result in S1012 that the image data received in S1008 is not the full screen size. The second determination is to obtain a determination result in S1013 that the image data received in S1008 is image data of a specific area. If neither of the processes in S1014 or S1027 described above has been performed, input information associated with area information indicating the display area of the VNC screen (first screen) for which the enlargement rate of the VNC screen has not been reset is transmitted. It turns out.

As explained above, in this embodiment, the following processing is executed. In other words, the ON/OFF setting of the reset function of the enlargement ratio of the VNC screen can be switched. Further, the enlargement ratio of the VNC screen is also reset when image data of a specific area is received, but depending on the type of soft key, the enlargement ratio of the VNC screen is not reset. As a result, the mobile terminal 100 displays the screen represented by the screen information of the operation screen (full screen) after screen transition received from the MFP 110 on the operation panel 206 only when a specific condition is satisfied. In this way, even if a screen transition occurs when one of the mobile terminal 100 and MFP 110 connected for remote operation is enlarged, a screen that does not impair the user's visibility can be displayed. Operability can be improved.

(Embodiment 3)
In the first embodiment, the case where the enlargement ratio of the VNC screen is reset when full-screen image data is transmitted from the MFP 110 to the mobile terminal 100 or when a soft key is pressed on the mobile terminal 100 is described. Indicated. However, although the operation screen displayed on the operation panel 3070 of the MFP 110 changes, image data of the entire screen size is not necessarily transmitted. As described above, since the mobile terminal 100 cannot accurately grasp the screen transition of the operation screen of the MFP 110, it is difficult to accurately reset the enlargement ratio of the VNC screen in accordance with the timing of the screen transition of the operation screen of the MFP 110. Can not.

Therefore, in this embodiment, the MFP 110 transmits notification information indicating that the operation screen has transitioned to the mobile terminal 100, and when the mobile terminal 100 receives the notification information, the magnification rate of the VNC screen is reset. explain. The basic configuration of this embodiment is the same as that of Embodiment 1, so the explanation will focus on the differences. It is assumed that the remote operation processing unit 520 of the MFP 110 generates screen information of the operation screen to which an attribute indicating whether the operation screen displayed on the operation panel 3070 has changed has been added. Alternatively, the remote operation processing unit 520 of the MFP 110 has a function of transmitting notification information indicating that the operation screen has changed to the VNC-connected mobile terminal 100 when the operation screen displayed on the operation panel 3070 has changed.

<Remote control processing using a mobile terminal>
Processing performed when the mobile terminal 100 remotely controls the MFP 110 according to the present embodiment will be described with reference to the drawings. Specifically, a process will be described in which the mobile terminal 100 establishes a VNC connection with the MFP 110 using the RFB protocol, resets the screen display and magnification ratio, and ends the VNC connection. FIGS. 11A and 11B are flowcharts showing the flow of processing executed by the mobile terminal 100 according to this embodiment. Each operation (step) shown in the flowcharts of FIGS. 11A and 11B is realized by the CPU 201 reading a program for realizing each control module stored in the ROM 202 or HDD 204 into the RAM 203 and executing it. Furthermore, some or all of the functions of the steps in FIGS. 11A and 11B may be realized by hardware such as an ASIC or an electronic circuit. Furthermore, since the basic configuration of this flowchart is the same process as the flowcharts of FIGS. 8A and 8B, the differences will be mainly explained. That is, S1101 to S1108 are the same as S801 to S808, and S1111 to S1116 are the same as S813 to S818, and their explanations will be omitted. S1119 to S1120 and S1122 to S1123 are the same as S821 to S822 and S826 to S827, and their explanation will be omitted.

In S1109, the CPU 201 determines whether notification information indicating that the operation screen has changed has been received from the MFP 110. If it is determined that the notification information has been received (YES in S1109), the process moves to S1110. If it is determined that the notification information has not been received (NO in S1109), the process moves to S1111.

In S1110, the CPU 201 resets the enlargement rate of the VNC screen. For example, if it is determined in S1109 that notification information indicating that the operation screen has changed has been received, the enlargement rate of the VNC screen will be reset.

In S1117, the CPU 201 determines whether the input operation received in S1112 is a drag. If it is determined that the input operation is a drag (YES in S1117), the process moves to S1118. If it is determined that the input operation is not a drag (NO in S1117), the process moves to S1121. In S1118, the CPU 201 determines whether the VNC screen is displayed in an enlarged manner. If it is determined that the VNC screen is displayed in an enlarged manner (YES in S1118), the process moves to S1119. If it is determined that the VNC screen is not displayed in an enlarged manner (NO in S1118), the process moves to S1121.

In S1121, the CPU 201 transmits input information indicating the content of the input operation accepted in S1112 to the MFP 110. For example, if it is determined in S1109 that notification information about a screen transition has been received and the process goes through S1110 and reaches S1121, the area information of the VNC screen (first screen) where the enlargement rate of the VNC screen has been reset is The input information associated with the will be sent. If the process of S1110 described above has not been performed, input information associated with area information of the VNC screen (first screen) whose enlargement ratio of the VNC screen has not been reset will be transmitted.

<Processing by MFP>
Processing executed by the MFP 110 according to the present embodiment will be described using figures. FIG. 12 is a flowchart showing the flow of processing executed by the MFP 110 according to this embodiment. Each step shown in the flowchart of FIG. 12 is realized by the CPU 3020 loading a control program stored in a memory such as the ROM 3030 into the RAM 3040 and executing it. Furthermore, since the basic configuration of this flowchart is the same process as the flowchart of FIG. 9, the differences will be mainly explained. That is, S1201 to S1204 are the same as S901 to S904, and S1207 to S1120 are the same as S905 to S908, and their explanations will be omitted.

In S1205, the CPU 3020 determines whether the operation screen displayed on the operation panel 3070 of the MFP 110 has undergone a screen transition. If it is determined that the screen has changed (YES in S1205), the process moves to S1206. If it is determined that the screen has not changed (NO in S1205), the process moves to S1207.

In S1206, the CPU 3020 notifies the mobile terminal 100 of notification information indicating that the operation screen has changed. Note that the notification as to whether the operation screen has transitioned may be sent together with the image data sent in S1204 by adding an attribute indicating whether the operation screen has transitioned.

As described above, in this embodiment, when the operation screen of the MFP changes, the portable terminal is notified of this, and the enlargement ratio of the VNC screen is reset based on the notification. As a result, the mobile terminal 100 displays the screen represented by the screen information of the operation screen (full screen) after screen transition received from the MFP 110 on the operation panel 206 only when a specific condition is satisfied. In this way, even if a screen transition occurs in an enlarged state on one of the mobile terminal 100 and MFP 110 connected for remote operation, a screen that does not impair the user's visibility can be displayed, and the operation can improve sex.

(Other embodiments)
The present disclosure is also realized by performing the following processing. That is, a program that implements one or more functions of the embodiments described above is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This can also be achieved by executing a process. In this case, the computer program and the storage medium storing the computer program constitute the present disclosure. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

301 Screen control unit 302 Communication unit 313 Remote control unit

Claims (16)

A program for causing a computer of a terminal communicating with an information processing device to execute a control method,
The control method includes:
a receiving step of receiving screen information corresponding to a screen displayed on a display unit of the information processing device from the information processing device;
a display step of displaying a screen based on the screen information received in the receiving step;
a changing step of changing the magnification ratio of the screen displayed in the display step;
including;
When the screen displayed on the display unit of the information processing device is transitioned and the screen information of the screen is received in the receiving step, the display step resets the magnification ratio of the screen to be displayed and displays the screen. Featured programs. including a determination step of determining whether the screen transitions;
2. The program according to claim 1, wherein when the determining step determines that the screen transitions, the receiving step receives the screen information in which the screen enlargement instruction has been reset. 3. The program according to claim 2, wherein if the screen size indicated by the screen information received in the receiving step is a full screen size, the determining step determines that the screen will transition. the information processing device has a hard key;
In the displaying step, soft keys imitating the hard keys of the information processing device and the screen are displayed in a superimposed manner,
3. The program according to claim 2, wherein when the screen information including operation information for the soft keys is received in the receiving step, it is determined in the determining step that the screen will transition. There are multiple hard keys,
If the receiving step receives the screen information including the operation information for a specific soft key among the plurality of soft keys, the determining step determines that the screen changes. The program described in 4. 6. When the receiving step receives the screen information including the operation information for a soft key other than the specific soft key, the determining step determines that the screen does not change. program. 3. The program according to claim 2, wherein when the screen information of a specific area on the screen is received in the receiving step, it is determined in the determining step that the screen has transitioned. including a setting step of setting a function to reset the enlarged display of the screen,
8. The program according to claim 2, wherein if the function is set in the setting step, the determining step determines that the screen will transition. 8. The program according to claim 2, wherein when notification information indicating that the screen will change is received in the receiving step, it is determined in the determining step that the screen will change. An attribute indicating whether the screen has transitioned is added to the screen information,
8. The program according to claim 2, wherein when the attribute added to the screen information indicates a transition of the screen, the determination step determines that the screen will transition. The program according to any one of claims 1 to 10, wherein the terminal is a smartphone or a tablet PC. The program according to any one of claims 1 to 11, wherein the information processing device is a multifunction device or an image forming device. A method for controlling a computer of a terminal communicating with an information processing device, the method comprising:
a receiving step of receiving screen information corresponding to a screen displayed on a display unit of the information processing device from the information processing device;
a display step of displaying a screen based on the screen information received in the receiving step;
a changing step of changing the magnification ratio of the screen displayed in the display step;
including;
When the screen displayed on the display unit of the information processing device is transitioned and the screen information of the screen is received in the receiving step, the display step resets the magnification ratio of the screen to be displayed and displays the screen. How to characterize it. A system comprising an information processing device and a terminal that communicates with the information processing device,
The information processing device includes:
comprising a transmitting means for transmitting screen information corresponding to a screen displayed on a display unit of the information processing device;
The computer of the terminal is
receiving means for receiving the screen information;
Display means for displaying a screen based on the screen information received by the receiving means;
changing means for changing the magnification ratio of the screen displayed by the display means;
has
When the screen displayed on the display unit of the information processing device changes and the screen information of the screen is received by the receiving means, the display means resets the magnification ratio of the screen to be displayed and displays the screen. system. An information processing device that communicates with a terminal,
transmitting means for transmitting screen information corresponding to a screen displayed on a display unit of the information processing device to the terminal;
display control means for causing the computer of the terminal to display a screen based on the screen information transmitted by the transmission means;
control means for causing the computer of the terminal to change the magnification ratio of the screen displayed by the display control means;
has
When the screen displayed on the display unit of the information processing device changes and the transmitting means transmits the screen information of the screen, the display control means resets the magnification ratio of the screen to be displayed and displays the screen. An information processing device characterized by: A method for controlling an information processing device that communicates with a terminal, the method comprising:
a transmitting step of transmitting screen information corresponding to a screen displayed on a display unit of the information processing device to the terminal;
a display control step of causing the computer of the terminal to display a screen based on the screen information sent in the sending step;
a control step of causing the computer of the terminal to change the magnification ratio of the screen displayed in the display control step;
including;
When the screen displayed on the display unit of the information processing device is changed and the screen information of the screen is transmitted in the transmission step, in the display control step, the enlargement ratio of the screen to be displayed is reset and the screen is displayed. A method for controlling an information processing device, characterized in that:

Images