JP6264813B2 - Operation support program, communication terminal, and processing device - Google Patents

Description

  The present invention relates to an operation support program, a communication terminal, and a processing device.

  For example, Patent Document 1 below proposes a technique for remotely operating a multifunction peripheral using a portable terminal having a touch panel superimposed on an LCD. Specifically, an image obtained by photographing the operation unit of the multifunction device is displayed on the LCD, and the multifunction device is caused to execute processing according to the operation of the touch panel. That is, of the images displayed on the LCD, for example, when a key image to which copy processing is assigned is designated, the portable terminal instructs the multifunction device to execute the copy processing.

JP2013-8183A

  In the case of the above technology, it is necessary to create a function for each multifunction device to instruct processing to the multifunction device according to the input operation to the touch panel of the mobile terminal, or when the multifunction device is upgraded, There was a problem that it was necessary to make additional changes.

  The present invention has been made to solve the above-described problems, and an object thereof is a display unit having a display area, a point on the display area, and a point designated by an input medium. A function that is executed by a communication terminal including an input unit that detects an indication point, a communication unit that communicates with the processing device, and a control unit, and that operates the processing device based on an input to the input unit has versatility. It is to provide an operation support program, a communication terminal, and a processing device that can be executed.

In order to achieve this object, the present invention includes a display unit having a display area, an input unit that detects a point on the display area and a point indicated by the input medium, and a process A process executed by the processing device in an operation support program executed by a communication terminal including a communication unit communicating with the device and a control unit and operating the processing device based on an input to the input unit. For an image that guides an input operation using the input medium for instructing, a transmission request unit that requests the processing device to transmit image data representing the image, and the transmission request unit that is transmitted in response to a request from the transmission request unit an image receiving means for receiving image data, a display control means for displaying an image represented by the received image data in the display area, the processing device And medium notification receiving unit replacement image receives the effect that gradually being replaced the serial image, the display control unit by the indicated point of the display area detected by the input unit in a state in which the image is displayed Input information transmitting means for transmitting the coordinate information and the type of input event for the input section to the processing device, and the control section executes the input information transmitting means. While receiving the information indicating that the image is being replaced, the transmission request unit does not transmit the type of the input event to the processing device, while the image requesting unit indicates that the image is being replaced. while receiving and are, it requests transmission of the image data.

In the operation support program according to the first aspect, the image receiving unit receives the image data from the processing device, and the display control unit displays the image represented by the image data in the display area. According to this image, it is possible to guide the input operation to the input unit of the communication terminal. Then, the coordinate information of the indication point detected by the input unit is transmitted to the processing device by the input information transmitting unit together with the type of the input event. Here, since the image data is transmitted from the processing device, the processing device can grasp in advance the correspondence between the position of the designated point in the image and the processing to be executed. In this case, the processing device can grasp what processing should be executed by grasping the position of the designated point in the image from the received coordinate information. For this reason, on the communication terminal side, it is not necessary to know what processing of the processing device is instructed by the input operation on the input unit. Therefore, versatility can be given to the function of operating the processing device based on the input to the input unit. In addition, since image data to be transmitted is received by the image receiving means in response to the transmission request means requesting transmission of image data, the image data is received when the communication terminal side needs the image data. can do. In addition, when changing the displayed image, the processing apparatus has a function of gradually replacing the image before the change with the image after the change, and the input information transmitting means receives that the image is being replaced. While the input event is not transmitted to the processing device, the transmission request means requests transmission of the image data while the fact that the image is being exchanged is received. Accordingly, it is possible to avoid a situation in which an input event is input from the terminal side and transmitted to the processing device side before the replacement of the image is completed on the processing device side. In addition, since the image data is requested to be transmitted while the image before the change is replaced with the image after the change, the change from the image before the change to the image after the change is also visually recognized on the display unit on the communication terminal side. can do.

  In the operation support program according to claim 2, paying attention to the fact that the image to be displayed in the display area of the display unit changes in accordance with the input operation to the input unit, the transmission processing by the input information transmitting unit is executed. Request transmission of image data as a condition. Thus, in addition to the effect of the operation support program according to the first aspect, the image data can be requested to be transmitted when the image to be displayed in the display area of the display unit is changed.

  The operation support program according to claim 3 requests the transmission of the image data on condition that a position fixing operation on the screen is transmitted as an input event. Here, when a position determination operation on the screen is performed as an input event, it tends to be appropriate to transition the image displayed in the display area of the display unit. Therefore, in addition to the effect of the operation support program according to the second aspect, there is an effect that the transmission of the image data can be requested at an appropriate timing.

  In the operation support program according to the fourth aspect, the image data transmission request is waited for a predetermined period when the images displayed in the display area of the apparatus-side display unit are gradually replaced. Thus, in addition to the effect of the operation support program according to claim 3, the situation where the image data is transmitted to the communication terminal in response to the image data transmission request before the replacement of the image is completed on the processing device side is avoided. There is an effect that can be.

In claim 5 operation support program, it does not send the type of input event that occurred between the image displayed on the display area of the device-side display unit that waiting a transmission request for image data to gradually be replaced . For this reason, in addition to the effect of the operation support program according to claim 3, the image displayed in the display area of the device-side display unit and the image displayed in the display area of the terminal-side display unit are inconsistent. A situation in which the processing device is operated based on an input event in a state can be avoided.

  In the operation support program according to the sixth aspect, the transmission of the image data is requested on condition that the displacement notification receiving means receives that the object in the image is displaced. When requesting transmission of image data when it is received that the object in the image is displaced, the image data representing the image in the middle of displacement of the object is received, and this is displayed in the display area of the terminal side display unit Can be displayed. Therefore, in addition to the effect of the operation support program according to claim 1 or 2, when an animation is displayed in the display area of the apparatus side display unit, the animation display can also be performed in the display area of the terminal side display unit. There is an effect.

  The operation support program according to claim 7, wherein when an object in the image displayed in the display area of the device-side display unit is displaced, the image displayed in the display area of the device-side display unit and the display of the terminal-side display unit Note that the image displayed in the region does not match. If the transmission process is executed based on the input operation on the terminal side input unit when the images do not match, there is a possibility that the operation of the processing device different from the user's intention is performed. In view of this point, in addition to the effect of the operation support program according to claim 6, in addition to the effect of the operation support program according to claim 6, by not performing the transmission processing to the processing device on condition that the displacement notification receiving unit receives that the displacement notification includes. There is an effect that it is possible to suitably avoid a situation where the processing apparatus is operated differently from the intention.

  9. The operation support program according to claim 8, wherein when an object indicating a position where an instruction point should be located in instructing a process to be executed by the processing apparatus is displaced in the display area of the apparatus side display section, the apparatus side display section Note that the position of the object is different between the display area and the display area of the terminal-side display unit. When an object is instructed by an operation on the terminal side input unit when the position of the object is different, the processing device side does not determine that the object intended by the user is instructed based on the coordinate information, and another part is instructed. There is a risk of being misrecognized as having been done. In this respect, since the transmission process is not executed when the object is displaced, in addition to the effect of the operation support program according to claim 7, it is preferable that the operation of the processing apparatus different from the user's intention is performed. There is an effect that it can be avoided.

  In the operation support program according to claim 9, when the progress degree of the processing executed by the processing device is displayed in an animation in the display area of the device-side display unit, this display itself indicates the processing executed by the processing device. Note that it does not guide the point where the indication point should be. In this case, by enabling transmission processing, in addition to the effect of the operation support program according to claim 7 or 8, the processing device can be operated when the progress degree animation is displayed. There is an effect that becomes possible.

  In the operation support program according to claim 10, on the condition that the displacement notification means receives that the displacement is included, the fact is displayed in the display area of the terminal side display unit. Therefore, in addition to the effect of the operation support program according to any one of claims 6 to 9, the fact that the object in the image displayed in the display area of the device-side display unit is displaced, There is an effect that it is possible to notify the user that the process resulting from the operation is being executed in the communication terminal.

  According to the communication terminal of the eleventh aspect, it is possible to achieve an effect according to the effect of the operation support program according to any one of the first to tenth aspects.

In the processing device according to the twelfth aspect, the image transmission means transmits the image data to the communication terminal on condition that the request reception means receives a transmission request for image data from the communication terminal. Thereafter, the input information receiving means receives the coordinate information of the indication point and the event type according to the input operation on the input unit of the communication terminal. Here, since the image data has been transmitted from the processing device, the processing device should know in advance the correspondence between the position of the indicated point and the type of input event in the image and the processing to be executed. Can do. In this case, it is possible to identify what processing should be executed by grasping the position of the designated point in the image from the received coordinate information. For this reason, the execution means can execute the process specified by the coordinate information and the type of the input event. As described above, in order to execute the process by the execution unit, the communication terminal only needs to transmit the coordinate information and the input event, so that it is not necessary to specify the process to be executed by the execution unit. For this reason, the versatility can be given to the function which operates a processing apparatus based on the input with respect to an input part. In addition, since the image data is transmitted on condition that the image data is requested, the image data can be transmitted when the communication terminal needs the image data. In addition, when changing the image, the image has a function of gradually replacing the image before the change with the image after the change. When the image is being gradually replaced, the fact is transmitted to the communication terminal. Thereby, on the communication terminal side, it is possible to perform processing according to the replacement of images.

  14. The processing apparatus according to claim 13, wherein the standby is performed even when the image data transmission request is received on condition that the object in the image displayed in the display area of the apparatus-side display unit is displaced. The transmission of image data by means is waited until the displacement is completed. Thus, in addition to the effect of the processing device according to claim 12, the state in which the image displayed in the display area of the device-side display unit and the image displayed in the display area of the terminal-side display unit do not match is continued. There is an effect that the time can be shortened as much as possible.

  In the processing device according to the fourteenth aspect, on the condition that the image being displayed in the display area of the device-side display unit includes an animation, the displacement notification means notifies the communication terminal of that fact. Thus, in addition to the effect of the processing device according to claim 12, the communication terminal can grasp that the image being displayed in the display area of the device-side display unit includes animation, and take measures accordingly. It becomes possible.

1 is a system configuration diagram according to a first embodiment. FIG. The figure which illustrates the change process of a screen. The figure which shows an image data map. The flowchart which shows the procedure of the process by the side of a portable terminal. The figure which shows the process at the time of starting an application in a portable terminal. 6 is a flowchart showing a procedure of image enlargement / reduction processing. The figure which shows the production | generation process of coordinate information. The flowchart which shows the procedure of an event reaction process. 6 is a flowchart showing a processing procedure on the MFP side. 9 is a flowchart showing a processing procedure on the MFP side according to the second embodiment. The flowchart which shows the procedure of the process by the side of a portable terminal. 10 is a flowchart showing a processing procedure on the MFP side according to the third embodiment. The flowchart which shows the procedure of the process by the side of a portable terminal. FIG. 10 is a diagram illustrating an image displayed on the MFP side according to the fourth embodiment.

<First Embodiment>
Hereinafter, a first embodiment according to the present invention will be described with reference to the drawings.

  FIG. 1 shows a system configuration according to the present embodiment. The illustrated multifunction peripheral device (MFP 10) has a printing function, a scanner function, a facsimile function, and the like. The MFP 10 includes a CPU 12, a flash memory 14, a RAM 16, a communication unit 18, an LCD 20, a touch panel 22, a scanner 24, a printer 26, and a modem 28, which are connected to each other via an input / output port 30. The CPU 12 controls each function of the MFP 10 and each unit connected to the input / output port 30 according to fixed values and programs stored in the flash memory 14 and data stored in the RAM 16.

  The flash memory 14 is a rewritable nonvolatile memory, and the RAM 16 is a rewritable volatile memory. The LCD 20 is a liquid crystal display device and displays various screens. A touch panel 22 is provided on the screen of the LCD 20 in an overlapping manner. The touch panel 22 is a user interface that outputs indication point information indicating a position on the touch panel 22 where an input medium such as a finger or a stick is touched or approached.

  Hereinafter, contact or approach to the touch panel 22 by the input medium is referred to as pressing of the touch panel 22 by the input medium, and an increase in the distance between the input medium and the touch panel 22 after pressing is referred to as lifting. Displacement of the pointing point, which is a point on the touch panel 22 that is touched or approached, by displacing the input medium without being lifted after being pressed is referred to as a drag operation. Note that the point on the touch panel 22 in this embodiment is the center point of the area on the touch panel 22 where the contact or approach of the input medium is detected, but is not limited thereto, and the contact or approach of the input medium is detected. An arbitrary point in the region on the touch panel 22 may be used, or the region itself may be read as a point in the present embodiment. These definitions also apply to the touch panel 46 of the mobile terminal 40.

  The scanner 24 reads a document and converts it into image data. The printer 26 prints an image based on the image data on a recording sheet. The modem 28 modulates image data to be transmitted at the time of FAX transmission into a signal that can be transmitted to a telephone line network (not shown), or transmits a signal input from a telephone line network (not shown) as image data. Demodulate. The communication unit 18 is an interface for performing wireless communication. In the present embodiment, the wireless communication performed by the communication unit 18 is Wi-Fi communication (Wi-Fi is a registered trademark) compliant with the IEEE802.11b / g standard.

  The mobile terminal 40 is, for example, a smartphone or a mobile phone. The portable terminal 40 includes a communication unit 42, an LCD 44, a touch panel 46, a voice input / output unit 48, a telephone network communication unit 50, a CPU 52, a flash memory 54, and a RAM 56, which are connected to each other via an input / output port 58. . The CPU 52 controls each unit connected to the input / output port 58 according to a fixed value, a program, or the like stored in the flash memory 54 or the like.

  The flash memory 54 is a rewritable nonvolatile memory, and the RAM 56 is a rewritable volatile memory. The LCD 44 is a liquid crystal display device and displays various screens. The touch panel 46 is a user interface that is provided so as to overlap the LCD 44 and outputs instruction point information indicating a position on the touch panel 46 where an input medium such as a finger or a stick is touched or approached.

  The voice input / output unit 48 is a voice input / output device including a microphone, a speaker, and the like. The telephone network communication unit 50 is a circuit for performing a call via a mobile phone network (not shown). The communication unit 42 is an interface for performing wireless communication. In the present embodiment, the wireless communication performed by the communication unit 42 is Wi-Fi communication (Wi-Fi is a registered trademark) compliant with the IEEE802.11b / g standard. The mobile terminal 40 is Wi-Fi connected to the MFP 10 having the communication unit 18 by an infrastructure mode via the communication unit 42 and an access point (not shown) as a relay device.

  The MFP 10 displays an image for guiding an input operation using an input medium on the display area 20a of the LCD 20, and performs facsimile transmission processing in response to an input operation on the touch panel 22, such as pressing, dragging, and lifting. In addition, various processes such as a printing process and a scanning process are executed. It is difficult to display all of the images including the object that guides the input operation for instructing the execution of these processes on the display area 20a of the LCD 20 at a time because of the size of the display area 20a. For this reason, in this embodiment, it is possible to switch the image displayed on the display area 20a of the LCD 20. 2A is an example of an image displayed on the display area 20a of the LCD 20, and FIG. 2C is another example of an image displayed on the display area 20a of the LCD 20. In FIG. 2A, an object for specifying a facsimile function (a button described as FAX), an object for specifying a copy function (button described as Copy), and an object for specifying a scan function (described as Scan). Button) is displayed. On the other hand, FIG. 2C shows an example in which an object for instructing execution of various processes related to the facsimile function, such as an object for instructing execution of the facsimile function (a button described as FAX Start), is shown.

  In the present embodiment, when the image displayed on the display area 20a of the LCD 20 is changed in the MFP 10, the display (at least the position of the object in the image displayed on the display area 20a is displaced at a speed that can be sensed by the user). Change the image using animation display. That is, when changing from the image before the change shown in FIG. 2A to the image after the change shown in FIG. 2C, the image is changed from the right side of the image before the change (entire screen) with the change of the image. Slide the image (entire screen) gradually. FIG. 2B shows a stage in the middle of sliding. During this halfway period, an input operation is not accepted on the touch panel 22 of the MFP 10.

  The change of the image in the display area 20a of the LCD 20 is basically performed in accordance with an input operation by the user. Specifically, the touch panel 22 is moved up after an object that causes screen transition is pressed. This is because, in the present embodiment, the pressing operation on the touch panel 22 means an operation for selecting the position of the designated point, and the lifting operation means an operation for confirming the position of the designated point. Note that the position of the pointing point that has been pressed is stored in the RAM 16.

  However, in the present embodiment, not only the lifting operation on the touch panel 22, but when the information to be notified to the user is generated inside the MFP 10, the object that causes the screen transition is pressed and not released. In addition, the screen of the display area 20a of the LCD 20 may be changed. Here, the information to be notified to the user includes, for example, a case where the ink remaining amount of the printer 26 has decreased. This screen transition is basically a pop-up display. Note that the pop-up display may be deleted from the display area 20a by being performed for a predetermined time.

  In this embodiment, each screen displayed in the display area 20a of the LCD 20 is given information (screen ID) for specifying the screen. As shown in FIG. 3, the screen ID is associated with an image by an image data map stored in the flash memory 14 of the MFP 10. The image data map is data in which a screen ID and information on necessity of storage are associated for each image data. Here, the information on necessity of storage will be described in detail in the description of the processing (S120) of FIG.

  In addition to the input operation on the touch panel 22, the MFP 10 can also execute various functions in response to instructions from the mobile terminal 40. In order to make this possible, in the present embodiment, an image for guiding an input operation using the input medium displayed on the display area 20a of the LCD 20 of the MFP 10 can be displayed also on the LCD 44 of the portable terminal 40. That is, when the MFP 10 transmits image data representing this image to the portable terminal 40, the portable terminal 40 displays the image represented by the received image data in the display area 44 a of the LCD 44. Thus, the user can perform an input operation using the input medium on the touch panel 46 in accordance with the image displayed in the display area 44a of the LCD 44. Then, the portable terminal 40 outputs the coordinate information of the indicated point on the display area 44a of the LCD 44 accompanying the input operation on the touch panel 46 to the MFP 10. Based on this coordinate information, as will be described in detail later with reference to FIG. 7, the MFP 10 can grasp which position of the image displayed on the display area 20 a of the LCD 20 is designated. Thus, for example, when the indication point is located in the object for instructing execution of the printing process in the display area 44a of the LCD 44, the MFP 10 recognizes that the execution of the printing process is instructed based on the coordinate information. Execute print processing. Thus, the MFP 10 can be operated by an input operation on the touch panel 46 of the portable terminal 40. At this time, the portable terminal 40 only notifies the MFP 10 of the coordinate information of the indication point and the type of input event (pressed, dragged, lifted) to the touch panel 46. In the portable terminal 40, the input to the touch panel 46 is performed. There is no need to know what processing execution instruction is for the MFP 10.

  Hereinafter, processing related to the operation of the MFP 10 by the mobile terminal 40 will be described in detail with reference to the drawings.

  FIG. 4 shows a procedure of processing executed by the mobile terminal 40. This process is realized by the CPU 52 executing an operation support program stored in the flash memory 54. Each process of S14 to S54 is executed each time an input operation on the touch panel 46 is received in the process of S14. That is, the processes from S14 to S54 are processes that can be executed in parallel.

  The CPU 52 executes the LCD data acquisition process when the operation support program (application) is started (S10). That is, as shown in FIG. 5, CPU 52 outputs a signal requesting image data (LCD data) representing an image displayed on display area 20 a of LCD 20 of MFP 10 to MFP 10. Then, the CPU 52 receives the LCD data transmitted from the MFP 10, and displays it in the display area 44 a of the LCD 44.

  Next, the CPU 52 receives additional information transmitted from the MFP 10 (S12). Here, the additional information includes a screen ID for specifying the image requested in S10. The MFP 10 includes the screen ID corresponding to the image data transmitted to the portable terminal 40 in the additional information using the image data map shown in FIG.

  Subsequently, the CPU 52 waits for an input operation on the touch panel 46 (S14). And when CPU52 receives input operation, it is judged whether the input event regarding the input operation is a pressing event (S16). When determining that the event is a pressing event (S16: Yes), the CPU 52 stores the position of the pointing point that has been pressed in the RAM 56, and determines whether or not an input event can be received on the touch panel 46 ( S18). This process is for determining whether or not the process of S46 described later is being performed. If the CPU 52 determines that it is not possible to accept an input event for the touch panel 46 (S18: No), the process proceeds to S14 in order to accept the input event when the input to the touch panel 46 can be accepted. . On the other hand, when the CPU 52 determines that the input event can be received with respect to the touch panel 46 (S18: Yes), the CPU 52 executes an image enlargement / reduction process (S20).

  FIG. 6 shows a procedure for image enlargement / reduction processing. In this process, the pressing event received in S14 is a pressing event that instructs the MFP 10 to execute the process, or one of the continuous pressing events received by performing a pinch-in / pinch-out operation by the input medium. This is for identifying whether the event is a press event. Here, the pinch-in operation is an operation of placing two fingers, for example, a thumb and an index finger on the touch panel and pinching with the two fingers, and is performed when the image is reduced. . The pinch-out operation is an operation of placing two fingers, for example, a thumb and an index finger on the touch panel, and pushing and spreading with the two fingers, for example, when enlarging an image. . Therefore, in the case of a pinch-in and pinch-out operation, it is an input operation that is irrelevant to the MFP 10 because it instructs to reduce or enlarge an image displayed in the display area 44 a of the LCD 44 of the portable terminal 40.

  The CPU 52 determines whether or not the waiting process described in step S61 has been executed (S60). When the CPU 52 determines that the waiting process is not being executed (S60: No), the CPU 52 starts the waiting process for waiting for a predetermined time (S61). Here, the predetermined time is set according to a time difference between the pair of pressing timings when two fingers as input media are pressed on the touch panel 46 so that the user performs a pinch-in operation or a pinch-out operation. Specifically, the time difference is set to a value that can normally be taken. For example, in this embodiment, it is set to 0.5 seconds. The CPU 52 determines whether another pressing event has occurred during the waiting process (S62). This process is for determining whether pinch-in or pinch-out is performed. When the CPU 52 determines that another pressing event has occurred during the waiting process (S62: Yes), the CPU 52 displays it in the display area 44a of the LCD 44 in accordance with the movement state of the pair of instruction points corresponding to the pair of pressing events. Processing for enlarging or reducing the image to be performed is executed (S64). In practice, it is desirable that the process of S62 is not executed after waiting for a predetermined time, but is repeatedly executed until an affirmative determination is made before the predetermined time elapses. Thereby, when an affirmative determination is made in S62, it is possible to quickly shift to the process of S64.

  On the other hand, when the CPU 52 determines that another pressing event does not occur during the waiting process (S62: No), the CPU 52 proceeds to the process of S22 in FIG.

  Thereafter, when the other input medium approaches or comes into contact with the touch panel 22 before the take-off event occurs, the CPU 52 makes an affirmative determination again in the process of S16 of FIG. 4 and proceeds to the image enlargement / reduction process. , It is determined that the waiting process of S61 is being executed (S60: Yes), and the process proceeds to S72. At this point, since there are two indication points, there is a possibility that the user is trying to pinch-in and pinch-out, but the indication points did not become two during the waiting process time. In the process of S72, it is discriminated whether the operation is a pinch-in / pinch-out operation or whether another finger happens to press the touch panel 46 during dragging.

  Specifically, the CPU 52 determines whether or not the drag storage count incremented by the process of S36 described later is equal to or less than a certain number (S72). The drag storage count indicates the sampling count when the drag operation is performed, and the drag storage count increases, indicating that the drag time is longer. Here, the fixed number is set to a value that is difficult to assume in the case of pinch-in and pinch-out. If the CPU 52 determines that the number exceeds a certain number (S72: No), the CPU 52 determines that the time for the drag operation is too long for the pinch-in / pinch-out operation, and proceeds to the process of S22 in FIG. On the other hand, if the CPU 52 determines that the number is less than a certain number (S72: Yes), it is considered that a pinch-in / pinch-out operation is performed, and therefore notifies the MFP 10 that the transmission of the pressing event is canceled (S74). ). This is because, when the CPU 52 makes an affirmative determination in S72, the user intends to perform a pinch-in operation or a pinch-out operation, but another one point is detected after one point is detected on the touch panel 46. Since it took time to be performed, a negative determination is made in the process of S62, and a press event is transmitted to the MFP 10 by the process of S24 through the process of S22 described later. This cancellation process is a premise of the process of S160 of FIG. When completing the process of S74, the CPU 52 executes an enlargement / reduction process according to the movement state of the pair of instruction points (S64).

  The CPU 52 determines whether or not all indication points are detected after the execution of the process of S64. In other words, it is determined whether all the input media have moved away from the touch panel 22 (S66). If it is determined that all indicated points are no longer detected (S66: No), a process of initializing the drag memory count is executed (S68), and the process of S14 of FIG. 4 is waited for a new input to the touch panel 46. Return to. Note that the drag storage count changes to the value after initialization by the processing of S68 is actually a case where the processing of S74 described later is executed. If an affirmative determination is made in S62, the processing of S68 is actually executed. Is not necessary.

  When the process of S20 in FIG. 4 is completed, that is, when a negative determination is made in S62 and S72, the CPU 52 executes a process of converting the pressed position (S22). For example, as shown in FIG. 7A, the coordinates (drawing start coordinates: X, Y) and the coordinates (A, B) of the pressed point in the display area 44a of the LCD 44 are used as a reference for displaying an image. If the difference (A−X, B−Y) is divided by the enlargement factor k of the displayed image, the value {(A−X) / k, (B−Y) / k} is calculated. Good. The difference (A−X, B−Y) is the coordinate of the pressed point when the drawing start coordinates (X, Y) are re-taken from the origin. Here, the enlargement factor k is a ratio of the size of the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 to the size of the image displayed on the display area 20a of the LCD 20 of the MFP 10. Here, the size is not limited to the actual length, and may be, for example, the number of display pixels. In this case, for example, when the area per pixel is smaller in the LCD 44 of the portable terminal 40 than in the LCD 20 of the MFP 10, the actual size of the displayed image is not limited even if the enlargement ratio is 1. What is displayed on the LCD 44 is smaller than what is displayed on the LCD 20.

  Even if the image is enlarged or reduced in the display area 44a of the LCD 44 of the portable terminal 40, the conversion process may be performed by any of the images displayed in the display area 20a of the LCD 20 in the MFP 10. This is intended to make it possible to grasp whether the point is a pressing point. Such an aim can also be realized by the ratio of the length from the reference point to the pressed point with respect to the length of the image in each component of the two-dimensional coordinates, as shown in FIG. 7B. In other words, the pair of end portions on the diagonal line of the image is the drawing start coordinate (X, Y) and the drawing end coordinate (Xe, Ye), and the length of the displayed image and the drawing for each of the two-dimensional coordinate components. The ratio of the length of the pressed point to the start coordinate {(AX) / (Xe-X), (BY) / (Ye-Y)} may be used.

  Subsequently, in the process of S24 of FIG. 4, the CPU 52 determines that the pressed position information obtained by the above process (the coordinate information of the pointing point instructed by pressing), the type of the input event is the pressed event, The screen ID of the image displayed in the display area 44a is transmitted to the MFP 10 (S24). Subsequently, the CPU 52 executes an event reaction process (S26). FIG. 8 shows this process.

  In this series of processing, the CPU 52 notifies that the screen ID of the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 is different from the screen ID of the image displayed on the display area 20a of the LCD 20 of the MFP 10. Is received from the MFP 10 (S80). This notification is made by the process of S140 of FIG. 9 described later, which is executed in the MFP 10 in accordance with the process of S24 of FIG. Here, when a pressing operation is performed on the touch panel 46 of the portable terminal 40 and a pressing event is transmitted to the MFP 10, the image in the display area 20 a of the LCD 20 of the MFP 10 does not change. This is because, as described above, in the present embodiment, the screen is changed by performing an operation for confirming the pointing point (a lifting operation). Nevertheless, the screen ID may differ, as described above, when information to be notified to the user is generated in the MFP 10, it is displayed in the display area 20 a of the LCD 20 regardless of the input operation by the user. This is because the image is changed.

  And CPU52 complete | finishes an event reaction process, when notification of a different effect is not received for the predetermined time (S80: No). On the other hand, when the CPU 52 receives a notification indicating that there is a difference (S80: Yes), the CPU 52 executes LCD data acquisition processing assuming that there is LCD data to be acquired from the MFP 10 (S82). This process is the same as the process of S10 in FIG. Subsequently, the CPU 52 executes additional information acquisition processing (S84). This process is the same as the process of S12 of FIG. 4, but in the process of S84, in addition to the screen ID, next screen presence / absence information is also acquired. Subsequently, the CPU 52 displays an image for notifying that the screen is updated because the screen ID is different in the display area 44a of the LCD 44 (S86). This may be realized by, for example, pop-up display. This process notifies the user that the screen is changed because the image displayed on the display area 20a of the LCD 20 of the MFP 10 and the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 do not match. It is made for that purpose.

  Next, the CPU 52 initializes the number of times of drag storage described later (S88). Then, the CPU 52 determines whether there is a next image (next image) to be acquired from the MFP 10 based on the additional information received by the process of S84 (S90). In this process, the mobile terminal 40 represents each of a plurality of images due to a situation in which the image displayed on the display area 20a of the LCD 20 is changed on the MFP 10 side independently of the mobile terminal 40. When image data is received in order and displayed in the display area 44a of the LCD 44, it is assumed that the additional information includes information indicating that there is a next image. This setting will be described in detail in the processing of S114 to S122 in FIG. When determining that there is no next image (S90: No), the CPU 52 ends the event reaction process. On the other hand, when determining that there is a next image (S90: Yes), the CPU 52 executes a waiting process (S92) and returns to the process of S82. Here, the waiting time of the waiting process is set to a length that allows the user to sufficiently check the image currently displayed in the display area 44 a of the LCD 44.

  When the event reaction process ends, the CPU 52 returns to the process of S14 in FIG. 4 to wait for a new input operation on the touch panel 46.

  When determining that the newly generated event is not a pressing event (S16: No), the CPU 52 determines whether or not the newly generated event is a drag event (S28). If the CPU 52 determines that the event is a drag event by detecting the displacement of the indicated point on the touch panel 46 (S28: Yes), can the input event be accepted for the touch panel 46 as in the process of S18 above? It is determined whether or not (S30). If the CPU 52 determines that it is not possible to accept an input event for the touch panel 46 (S30: No), the CPU 52 returns to S14 to accept this when the input to the touch panel 46 is possible. On the other hand, when the CPU 52 determines that the input event can be received with respect to the touch panel 46 (S30: Yes), the CPU 52 executes a drag position conversion process (S32). In this process, the position of the current indication point is determined by the LCD 20 of the MFP 10 regardless of the size of the image displayed in the display area 44a of the LCD 44 in the manner of the process shown in FIG. 7A or 7B. It is a process which converts into the value (coordinate information) which can be specified as a position on the image displayed on the display area 20a.

  Subsequently, the CPU 52 transmits to the MFP 10 that the type of event that has newly occurred is a drag (drag event) and the coordinate information of the pointing point that accompanies the drag calculated by the process of S32 (S34). Subsequently, the CPU 52 executes a process of storing the number of drag events indicating the number of drag position samplings in the RAM 56 in S32 (S36). This process is a prerequisite for the process of S72 of FIG. Then, when completing the process of S36, the CPU 52 returns to the process of S14 to wait for a new input to the touch panel 46.

  On the other hand, when the CPU 52 determines that the newly generated event is not a drag event (S28: No), the CPU 52 determines whether the newly generated event is a lift event (S38). If the CPU 52 determines that the event is not a lift event (S38: No), the CPU 52 returns to the process of S14 in order to wait for a new input to the touch panel 46. On the other hand, when determining that the event is a lift event (S38: Yes), the CPU 52 determines whether or not an input event can be received with respect to the touch panel 46, similarly to the process of S18 (S40). . If the CPU 52 determines that it is not possible to accept an input event for the touch panel 46 (S40: No), the CPU 52 returns to the process of S14 to accept this when the input to the touch panel 46 is acceptable. . On the other hand, when the CPU 52 determines that the input event can be received with respect to the touch panel 46 (S40: Yes), the CPU 52 executes a lift position conversion process (S42). In this process, the position where the pointing point disappears, in other words, the position of the last detected pointing point (pressed position or drag position) is determined in the manner shown in FIG. 7A or 7B. This is a process of converting to a value that can be specified as a position on the image displayed on the display area 20a of the LCD 20 of the MFP 10 regardless of the size of the image displayed on the display area 44a of the LCD 44.

  Subsequently, the CPU 52 indicates that the newly generated event type is a lift, the screen ID of the image displayed on the LCD 44, and the lift position information calculated in the process of S42 (disappears with the lift). The coordinate information of the designated point) is transmitted to the MFP 10 (S44). Subsequently, the CPU 52 starts a process for prohibiting event reception (S46), and executes a waiting process (S48). Then, when the waiting process is completed, the CPU 52 executes an LCD data acquisition process that is the same process as the process of S10 (S50), and executes an additional information acquisition process that is the same process as the process of S12 (S52). ). In the process of S52, the next screen presence / absence information is acquired in addition to the screen ID. The waiting process is for waiting for transmission of a request signal for LCD data for a time required for changing the image displayed on the display area 20a of the LCD 20 of the MFP 10 due to the take-off event. As described above with reference to FIG. 2, this is based on the premise that an animation is displayed when an image displayed in the display area 20a of the LCD 20 is changed.

  If LCD data is requested from the portable terminal 40 to the MFP 10 during the animation display period, the image displayed on the MFP 10 may not match the image displayed on the portable terminal 40. In such a case, if an input operation is accepted, the MFP 10 may malfunction. That is, during the period in which the LCD data being animated is transmitted to the mobile terminal 40 and the image represented by the data is displayed in the display area 44a of the LCD 44, the image displayed on the display area 20a of the LCD 20 of the MFP 10 is changed. When completed, the MFP 10 can already perform an input operation on the touch panel 22. At this time, on the portable terminal 40 side, an input operation is performed on the touch panel 46 based on the image shown in FIG. 2B displayed in the display area 44a of the LCD 44, and the coordinate information of the indicated point is transmitted to the MFP 10. Then, the MFP 10 may erroneously recognize that an input operation has been performed on a location different from the location intended by the user in the image displayed on the display area 20a of the LCD 20. If the misrecognized indication point is located in the object indicating the location where the indication point should be in order to instruct the processing executed by the MFP 10 in the display area 20a of the LCD 20, the MFP 10 may malfunction. . Therefore, the processing of S46 and S54 in FIG. 4 prevents the input operation to the touch panel 46 of the portable terminal 40 during the animation display period on the MFP 10 side.

  When completing the process of S52, the CPU 52 cancels the event reception prohibition process (S54), and returns to the process of S14 to wait for a new input to the touch panel 46. Incidentally, it is prohibited to execute this process at the end of the process of S48 on the condition that the image displayed in the display area 44a of the LCD 44 is changed to the image displayed in the display area 20a of the LCD 20. This is for canceling the processing.

  FIG. 9 shows a processing procedure on the MFP 10 side. This process is executed by the CPU 12 of the MFP 10.

  The CPU 12 executes standby processing such as the type of event and coordinate information transmitted from the mobile terminal 40 (S100). When an event occurs, the CPU 12 determines whether the event is a request for LCD data (S102). If the CPU 12 determines that an event for requesting LCD data has occurred (S102: Yes), the CPU 12 determines whether there is stored LCD data (S104). In this process, as described above, the image displayed on the display area 20a of the LCD 20 of the MFP 10 generates information to be notified to the user in the MFP 10 regardless of the input operation to the touch panel 46 of the portable terminal 40. It is provided in view of the fact that it may be changed. That is, in this case, image data that has not yet been transmitted to the portable terminal 40 is stored in the RAM 16 by the process of S122 described later. Therefore, in the process of S104, an image representing the image being displayed on the display area 20a of the LCD 20 In addition to the data, it is determined whether there is image data stored in the RAM 16. When determining that the stored LCD data does not exist (S104: No), the CPU 12 acquires a screen ID that is an ID of the image data displayed in the display area 20a of the LCD 20 (S106). This process can be executed based on the image data map shown in FIG. Subsequently, the CPU 12 stores the acquired screen ID in the RAM 16 as a transfer screen ID to be transferred to the portable terminal 40 (S108). Then, the CPU 12 transfers the LCD data and the transfer screen ID as additional information to the portable terminal 40 (S110). Note that the additional information transfer process is actually executed in response to a request for additional information from the portable terminal 40 in addition to the LCD data request by the processes of S12, S52, and S84 in FIG. The

  On the other hand, when determining that the event is not an LCD data acquisition instruction event (S102: No), the CPU 12 determines whether or not an event that causes a transition of the screen of the LCD 20 has occurred (S116). This process is for determining whether or not an event has occurred in which the screen of the LCD 20 changes despite no instruction from the portable terminal 40. The screen transition in this case is a pop-up display or the like made when information to be notified to the user is generated in the MFP 10 as described above. Here, in the image data map of FIG. 3, different screen IDs are assigned when the pop-up display is performed and when the pop-up display is not performed.

  When determining that a screen transition event has occurred (S116: Yes), the CPU 12 determines whether or not the transition source screen ID matches the transfer screen ID stored in S108 (S118). This process is for determining whether the image before the transition has not yet been transferred to the portable terminal 40. When determining that they do not match (S118: No), the CPU 12 determines whether the screen needs to be stored (S120). This process is executed by referring to the image data map shown in FIG. That is, whether or not storage is necessary is determined in advance in the image data map, and based on this, it is determined whether or not storage is necessary. And CPU12 preserve | saves LCD data in RAM16, when a memory | storage is required (S120: Yes) (S122). Then, the CPU 12 saves the LCD data, determines that they match in S118 (S118: Yes), and further determines that the screen does not need to be stored (S120: No), the process of changing the screen is performed. Execute (S124) and shift the process to S100 in order to wait for a new event to occur.

  If the CPU 12 determines that the LCD data stored in the process of S104 is present (S104: Yes), the CPU 12 acquires the stored LCD data (S112). Next, the CPU 12 transfers the LCD data and the additional information (S114), and returns to the process of S100 to wait for a new event. Here, the additional information includes information indicating that there is a next image. This is a setting for causing the portable terminal 40 to transmit a signal requesting LCD data again. That is, in the present embodiment, when the LCD data request is issued from the mobile terminal 40, the setting is such that the LCD data is transferred from the MFP 10 to the mobile terminal 40, and the MFP 10 transmits the request to the mobile terminal 40 without receiving the request. Does not voluntarily transmit LCD data. Here, the situation where there is stored LCD data is a situation where a negative determination is made in S118 and an affirmative determination is made in S120, which is a situation that cannot be grasped on the portable terminal 40 side. For this reason, information indicating that there is a next image is added as additional information. Accordingly, when the CPU 52 makes an affirmative determination in S90 of FIG. 8, the CPU 52 requests LCD data again by the processing of S82. Note that the additional information transfer process in the process of S110 actually requires additional information from the portable terminal 40 separately from the LCD data request by the processes of S12, S52, and S84 in FIG. Will be executed accordingly.

  When determining that the newly generated event is neither a request for LCD data nor a transition of the main body screen (S116: No), the CPU 12 determines whether or not it is a pressing event (S126). If the CPU 12 determines that the event is a pressing event (S126: Yes), the CPU 12 executes a process of acquiring (receiving) the screen ID added to the event (S128). This process is based on the process of S24 in FIG. Subsequently, the CPU 12 determines whether or not the acquired screen ID matches the screen ID of the image displayed in the display area 20a of the LCD 20 (S132). This process is for determining whether or not the image displayed in the display area 20a of the LCD 20 of the MFP 10 matches the image displayed in the display area 44a of the LCD 44 of the portable terminal 40. If the CPU 12 determines that they match (S132: Yes), the CPU 12 executes the processes of S134 to S138, assuming that the pressed event notified from the portable terminal 40 is valid as an instruction to the MFP 10. That is, the CPU 12 first performs a process of converting the pressed position data if necessary (S134).

  Subsequently, the CPU 12 stores the pressed position in the RAM 16 (S136). This process is a premise of the process of S164 described later. Then, the CPU 12 executes a process accompanying the pressing (pressing process) (S138), and returns to the process of S100 in order to wait for a newly occurring event. This processing is, for example, processing for visually displaying that an object is selected when an object is present at the pressed position. However, the display area 20a of the LCD 20 may not be changed at all and may be processing only within the MFP 10.

  In contrast, if the CPU 12 determines that the screen IDs do not match in the process of S132 (S132: No), the CPU 12 notifies the mobile terminal 40 that the screen IDs do not match (S140), and waits for a new event to occur. Return to the process. The process of S140 is a premise of the process of S80 of FIG. In this case, the pressing process (S138) is not executed. This is because when the image displayed on the display area 20a of the LCD 20 of the MFP 10 and the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 do not match, the pressing event notified from the portable terminal 40 is the MFP 10. This is because it is considered that it is not effective as an instruction for processing.

  When determining that the newly generated event is not a pressing event (S126: No), the CPU 12 determines whether the event is a drag instruction event (S142). If the CPU 12 determines that the event is a drag instruction event (S142: Yes), the CPU 12 executes a process of converting the drag position data as necessary in the process of S134 (S144). Subsequently, the CPU 12 determines whether or not a display corresponding to the drag, for example, a display for moving the dragged image, needs to be displayed on the LCD 20 of the MFP 10 in accordance with the drag operation on the touch panel 46 of the portable terminal 40 (S146). . This process can be executed by storing a value that determines whether or not it is necessary in advance in the flash memory 54 of the MFP 10. Whether or not the display according to the drag is necessary may be determined at the time of shipment of the MFP 10 or may be selectable by the user. If the CPU 12 determines that an image being dragged is necessary (S146: Yes), the CPU 12 displays a display corresponding to the drag on the LCD 20 of the MFP 10 in accordance with the drag operation on the touch panel 46 of the portable terminal 40 (S148). On the other hand, if the CPU 12 determines that the image being dragged is not necessary (S146: No), the CPU 12 executes processing for saving the current indication point accompanying the dragging in the RAM 16 (S150), and waits for a new event to occur. Therefore, the process returns to S100.

  When determining that the newly generated event is not a drag instruction event (S142: No), the CPU 12 determines whether the event is a lift instruction event (S152). If the CPU 12 determines that the event is a lift instruction event (S152: Yes), the CPU 12 determines whether or not the position of the instruction point that has changed from the time of pressing is saved by the process of S150 (S154). . When the CPU 12 determines that the position of the changed indication point is stored (S154: Yes), the CPU 12 executes a process of displacing the display of the indication point in the display area 20a of the LCD 20 to the stored position ( S156). Then, when this process is completed or when it is determined that the position of the changed indication point is not stored (S154: No), the CPU 12 executes a lift-off process (S158) and waits for a newly generated event. Therefore, the process returns to S100. The lifting process is a process executed in the MFP 10 in accordance with the lifting operation on the touch panel 46 of the portable terminal 40. For example, if the selection of the object is confirmed by performing a lifting operation on the object displayed on the display area 44a of the LCD 44 of the portable terminal 40, a visual display corresponding to this is displayed. It may be executed on 20a.

  When the CPU 12 determines that the newly generated event is not a take-off instruction event (S152: No), the CPU 12 determines whether the event is a press cancel event by the process of S74 of FIG. 6 (S160). If the CPU 12 determines that the event is not a press cancel event (S160: No), the CPU 12 returns to the process of S100 to wait for a newly generated event. On the other hand, when determining that the event is a press cancel event (S160: Yes), the CPU 12 determines whether a drag event has occurred after the press operation (S162). If it is determined that a drag event has occurred (S162: Yes), a process of returning the display position of the indicated point on the LCD 20 and the position of the object displaced along with this to the pressed position is executed (S164). This is for returning the display position to the pressed position first when the pressed event is canceled after the display position of the object selected by the pressing is changed by the drag operation continuing after the pressed operation. When this process is completed or when it is determined that there is no drag event (S162: No), a press cancel process is executed (S166), and the process returns to S100 to wait for a new event to occur. Here, for example, when the display indicating that the pressed object is selected, the pressing cancellation process means stopping the visual display.

  According to the embodiment described above, the following effects can be obtained.

  (1) The image displayed on the display area 20a of the LCD 20 of the MFP 10 is displayed on the display area 44a of the LCD 44 of the portable terminal 40. As a result, the input operation on the touch panel 46 of the portable terminal 40 can be performed in a manner similar to the input operation on the touch panel 22 of the MFP 10.

  (2) The coordinate information ((x, y) in FIG. 7 (a) or (m, n) in FIG. 7 (b)) determined according to the input operation on the touch panel 46 of the portable terminal 40 is converted into the MFP 10 (S24, S34, S44 in FIG. 4). Accordingly, since it is not necessary for the portable terminal 40 to know what processing of the MFP 10 is to be performed by the input operation on the touch panel 46, the function of operating the MFP 10 based on the input on the touch panel 46 is provided. It can have versatility.

  (3) The screen ID is associated with the image data (FIG. 3), the screen ID is received when the portable terminal 40 receives the LCD data, and the screen ID is transmitted when the coordinate information or the like is transmitted to the MFP 10. . As a result, the MFP 10 can grasp whether or not the image displayed in the display area 44 a of the LCD 44 of the portable terminal 40 matches the image displayed in the display area 20 a of the LCD 20 of the MFP 10. Therefore, the MFP 10 can deal with a mismatch when it occurs.

  (4) When it is determined that the image displayed in the display area 44a of the LCD 44 of the portable terminal 40 and the image displayed in the display area 20a of the LCD 20 of the MFP 10 do not match (S132: No) The terminal 40 side requested LCD data (S82). Thereby, the situation which is not in agreement can be eliminated.

  (5) When it is determined that the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 and the image displayed on the display area 20a of the LCD 20 of the MFP 10 do not match (S132: No), the MFP 10 The LCD data is received and the display is updated, and the fact that the image is updated due to the mismatch is displayed (S86). Thereby, the user can be made aware of the situation.

  (6) Additional information including a screen ID and LCD data are requested and received separately. As a result, the LCD data (for example, PDF data) transmitted from the MFP 10 can be converted (for example, converted to JPEG) in a state where no other data is attached. Compared with the case where data is combined, the conversion process can be performed easily.

  (7) An input operation on the touch panel 46 is not accepted until the LCD data is requested and received and displayed in the display area 44a of the LCD 44 (the process of S10 in FIG. 4). (The point which provided S14 later, the point which transfers to the process of S14 with the completion | finish of the process of S46-S54, S26). For this reason, the input operation of the touch panel 46 is not accepted in a period in which the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 and the image displayed on the display area 20a of the LCD 20 of the MFP 10 are inconsistent. Thus, it is possible to preferably avoid a situation in which the MFP 10 erroneously recognizes an instruction different from the instruction intended by the user.

  (8) Information on the presence or absence of the next image is included in the additional information. Accordingly, the LCD data can be appropriately requested from the portable terminal 40 side in the setting for transmitting the LCD data from the MFP 10 side to the portable terminal 40 side on the condition that there is a request from the portable terminal 40 side.

  (9) The request for transmission of LCD data from the mobile terminal 40 is a condition for the MFP 10 to transmit LCD data to the mobile terminal 40. Thereby, the timing at which the LCD data is transmitted from the MFP 10 to the portable terminal 40 can be set to the timing at which the portable terminal 40 needs the LCD data. Further, even when there are a plurality of portable terminals 40 that communicate with the MFP 10, it is possible to appropriately transmit the LCD data to the portable terminals 40 that require the LCD data.

  (10) An image representing an image before the transition on the condition that the screen transitions in a state where the image data of the image displayed on the display area 20a of the LCD 20 of the MFP 10 has not yet been transmitted to the portable terminal 40 The data was stored in the RAM 16 (S122). Thereby, the image data of the image before the transition can be transmitted to the portable terminal 40.

  (11) When the lifting operation is performed on the touch panel 46 of the portable terminal 40 and the lifting operation is transmitted to the MFP 10 as the type of the input event, transmission of LCD data is requested (S50 in FIG. 4). Here, the lifting operation is an operation for confirming the selection of the position on the image, whereby the image displayed on the display area 20a of the LCD 20 of the MFP 10 tends to change. Therefore, LCD data can be requested at an appropriate timing.

  (12) Prior to requesting the transmission of the LCD data in response to the lifting operation on the touch panel 46 of the portable terminal 40, the mobile terminal 40 waits for a predetermined period (S48 in FIG. 4). Thus, it is possible to avoid a situation in which the LCD data is transmitted to the portable terminal 40 in response to the LCD data transmission request from the portable terminal 40 before the image replacement is completed on the MFP 10 side.

  (13) The coordinate information ((x, y) in FIG. 7A, (m, n) in FIG. 7B) transmitted from the portable terminal 40 to the MFP 10 is displayed in the display area 44a of the LCD 44. Regardless of whether the image is enlarged or reduced, normalization information for specifying the position of the pointing point when the image has a specified size is included. Therefore, regardless of whether the image displayed in the display area 44a of the LCD 44 is enlarged or reduced, in the MFP 10, the input operation on the touch panel 46 of the portable terminal 40 instructs execution of any process. Can be grasped.

  (14) The coordinate information transmitted from the portable terminal 40 to the MFP 10 is information integrated with the standardized information. Thereby, the coordinate information transmitted from the portable terminal 40 to the MFP 10 can be made simple information.

  (15) When there are a plurality of designated points for the display area 44a of the LCD 44 of the portable terminal 40, the type of input event and coordinate information are not transmitted to the MFP 10 (in the case of Yes in S62 of FIG. 6). Configuration). As a result, a situation in which unnecessary information is transmitted to the MFP 10 can be avoided.

  (16) If another pressing event occurs again after the pressing event is detected and before the lift event occurs, a pressing event cancel event is transferred to the MFP 10 (S74 in FIG. 6). As a result, when the input operation on the touch panel 46 of the portable terminal 40 instructs the enlargement or reduction process, it is possible to avoid a situation where erroneous information about the input event is reflected in the MFP 10.

  (17) The lift event after the cancellation event of the pressing event is transferred (S74 in FIG. 6) is not transferred to the MFP 10 (settings in S66 and S68 in FIG. 6). Thereby, it is possible to avoid a situation where unnecessary input event information is transmitted to the MFP 10.

  (18) When a press cancel event is received, cancel processing is executed (S166 in FIG. 9). Thus, even if the type of the input event transmitted from the MFP 10 is later found to be inappropriate, this can be dealt with.

  (19) When a push cancel event is received and a drag event has occurred after the cancel event to be canceled, processing is performed to return the indicated point to the position when the press event was made (FIG. 9). S164). Thereby, on the LCD 20 of the MFP 10, the object moved by the drag can be returned to the original position. For this reason, the object can be displayed at the position before the pressing event when the pressing cancellation process (S166) is performed.

<Second Embodiment>
Hereinafter, the second embodiment will be described with reference to the drawings with a focus on differences from the first embodiment.

  In the first embodiment, in view of the fact that an animation is displayed when the image displayed on the LCD 20 of the MFP 10 is changed, a waiting process is performed in response to a take-off event on the mobile terminal 40 side (S48 in FIG. 4). Was provided. In contrast, in the present embodiment, when LCD data is requested from the portable terminal 40 to the MFP 10, when the animation display is performed on the MFP 10 side, the LCD data from the MFP 10 to the portable terminal 40 is displayed until the animation display is completed. Does not execute the transfer process.

  FIG. 10 shows changes to the processing shown in FIG. 9 in the processing procedure on the MFP 10 side according to the present embodiment. This process is executed by the CPU 12 of the MFP 10.

  If the CPU 12 determines in the process of S104 that the stored LCD data does not exist (S104: No), it is determined whether or not an animation is being displayed when changing the image displayed in the display area 20a of the LCD 20. Judgment is made (S170). If the CPU 12 determines that the animation is not being displayed (S170: No), the process proceeds to S106. On the other hand, when determining that the animation is being displayed (S170: Yes), the CPU 12 transmits an animation notification indicating that the animation is being displayed to the portable terminal 40 (S172). Subsequently, the CPU 12 waits until the animation display ends (S176). If the CPU 12 determines that the animation display has ended (S176: Yes), the CPU 12 notifies the mobile terminal 40 to that effect (S178), and proceeds to the processing of S106.

  FIG. 11 shows a processing procedure on the portable terminal 40 side. This process is realized by the CPU 52 executing a remote operation program stored in the flash memory 54. Note that, in FIG. 11, the same step numbers are attached for convenience to those corresponding to the processing shown in FIG. 4.

  In the present embodiment, when the process of S12 is completed, the CPU 52 determines whether or not an animation notification is made from the MFP 10 (S180). This process is based on the processes of S172 and S178 in FIG. If the CPU 52 determines that no animation notification has been made (S180: No), the process proceeds to S14. On the other hand, when the CPU 52 determines that the animation notification is made (S180: Yes), the CPU 52 prohibits the input operation reception (event reception) to the touch panel 46 and displays the fact on the display area 44a of the LCD 44 ( S182), the process proceeds to S184. In S <b> 182, the input operation may be accepted and the processing corresponding to the accepted input operation may not be executed even if the acceptance of the input operation on the touch panel 46 is not prohibited. In S184, the CPU 52 stands by until an animation end notification is received from the MFP 10 (S184: No). When an animation end notification is received (S184: Yes), the process proceeds to S10.

  In the present embodiment, the process proceeds to S180 even when a negative determination is made in S18, S30, or 40, or when the processes in S26, S36, and S52 are completed.

  According to the present embodiment described above, the following effects can be obtained in addition to the effects (1) to (4) and (7) to (19) of the first embodiment.

  (20) When the LCD data is requested from the MFP 10, the transmission of the LCD data is waited until the animation display is finished (S174 in FIG. 10). As a result, the duration of the state in which the image displayed on the display area 20a of the LCD 20 of the MFP 10 and the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 do not match can be shortened as much as possible.

  (21) When the notification that the animation is in progress is received from the MFP 10, the portable terminal 40 is configured not to accept an input operation on the touch panel 46. As a result, when the image displayed on the display area 20a of the LCD 20 of the MFP 10 and the image displayed on the display area 44a of the LCD 44 of the portable terminal 40 do not match, an input operation on the touch panel 46 is not accepted. Thus, it is possible to avoid a situation in which the MFP 10 is instructed against the user's intention.

<Third Embodiment>
Hereinafter, the third embodiment will be described with reference to the drawings with a focus on differences from the first embodiment.

  In the first embodiment, no animation is displayed on the display area 44a of the LCD 44 of the portable terminal 40 even when an animation is displayed in response to the image change in the display area 20a of the LCD 20 of the MFP 10. On the other hand, in the present embodiment, the animation display is executed also in the display area 44a of the LCD 44 of the portable terminal 40.

  FIG. 12 shows a processing procedure on the MFP 10 side according to the present embodiment. This process is executed by the CPU 12 of the MFP 10. Note that, in FIG. 12, the same step numbers are assigned for convenience to those corresponding to the processing shown in FIG.

  As illustrated, in the present embodiment, when the CPU 12 determines in the process of S104 that the stored LCD data does not exist (S104: No), the CPU 12 changes the image displayed in the display area 20a of the LCD 20. It is determined whether the accompanying animation is being displayed (S170). If the CPU 12 determines that the animation is not being displayed (S170: No), the process proceeds to S106. On the other hand, when determining that the animation is being displayed (170: Yes), the CPU 12 transmits an animation notification indicating that the animation is being displayed to the portable terminal 40 (S190).

  FIG. 13 shows a processing procedure on the portable terminal 40 side. This process is realized by the CPU 52 executing a remote operation program stored in the flash memory 54. In FIG. 13, the same step numbers are assigned for convenience to those corresponding to the processing shown in FIG.

  In the present embodiment, when the process of S52 is completed, the CPU 52 determines whether an animation notification has been made (S200). This process is premised on the process of S190 in FIG. If the CPU 52 determines that no animation notification has been made (S200: No), the CPU 52 proceeds to S54. On the other hand, if the CPU 52 determines that an animation notification has been made (S200: Yes), the CPU 52 returns to the process of S50 and executes the LCD data acquisition process again.

  According to the present embodiment described above, the following effects can be obtained in addition to the effects (1) to (4) and (7) to (19) of the first embodiment.

  (22) When an animation is displayed in the display area 20a of the LCD 20 of the MFP 10, the LCD data is requested again from the portable terminal 40 side. Thereby, animation display can also be performed in the display area 44 a of the LCD 44 of the portable terminal 40.

<Fourth Embodiment>
Hereinafter, the fourth embodiment will be described with reference to the drawings with a focus on differences from the first embodiment.

  In the present embodiment, a progress bar display that displays the progress of processing is executed as an animation display in the display area 20a of the LCD 20 of the MFP 10. FIG. 14 shows an example in which a progress bar 70 indicating the degree of progress of the printing process is displayed in the display area 20 a of the LCD 20 of the MFP 10. The progress bar 70 changes the display in accordance with the progress of the printing process so as to display the progress of the printing process. In FIG. 14, FIG. ) Shows an example of displaying that the printing process is in progress. In addition, in order to change the display of the progress bar 70 in accordance with the progress of the printing process also on the display area 44a of the LCD 44 of the portable terminal 40, information indicating that the progress bar 70 is displayed is included in the additional information. After the process of S54 in FIG. 4, LCD data may be periodically requested from the portable terminal 40 to the MFP 10.

  A cancel button 72 for canceling the printing process is displayed in the display area 20a of the LCD 20 of the MFP 10 before the printing process is completed. For this reason, the cancel button 72 is displayed also in the display area 44a of the LCD 44 of the portable terminal 40 by the acquisition process of the LCD data. When the cancel button 72 is pressed and released by an input operation on the touch panel 46 of the mobile terminal 40, the MFP 10 stops the printing process. Thereby, it is possible to instruct the mobile terminal 40 to cancel the printing process.

<Fifth Embodiment>
Hereinafter, the fifth embodiment will be described focusing on differences from the second embodiment.

  In the second embodiment, when the animation display is performed on the MFP 10 side, the mobile terminal 40 is configured not to transmit the touch panel event to the MFP 10 until the animation display is completed. However, in the fifth embodiment, Only in the case of a specific animation, the touch panel event is transmitted.

  Specifically, in S180 of FIG. 11 showing the second embodiment, when it is determined that the animation notification is made (S180: Yes), the CPU 52 displays on the MFP side based on the additional information acquired in S12. If it is determined that the type of animation being performed is the progress bar shown in FIG. 14 described above, the process proceeds to S14, and if it is determined that the animation is not a progress bar, the process proceeds to S182. To do. Note that the MFP 10 includes information specifying the type of animation displayed on the MFP 10 side in the additional information transmitted to the mobile terminal 40.

<Correspondence between each means and the embodiment>
Display unit (terminal side display unit) ... LCD 44, device side display unit ... LCD 20, input unit (terminal side input unit) ... touch panel 46, device side input unit ... touch panel 22, communication unit ... communication unit 42, control unit ... CPU 52, Processing unit: MFP 10, storage unit: RAM 16, transmission request means: S10, S50, S82, image reception means: S10, S50, S82, display control means: S10, S50, S82, input information transmission means: S24, S34, S44 displacement notice receiving means ... S 18 0, displacement notifying means ... S182, S190, request receiving means ... S102, the image transmitting means ... S110, the input information receiving means ... S100, execution unit ... S138, S148, S158, waiting means ... S174 , Image replacement notification receiving means ... S200,
Selection of the position on the image as an input event ... Pressing operation Determination of the position on the image as an input event ... Leaving operation Function for gradually replacing the image after the change ... FIG.
Animation display of progress ... FIG.

<Other embodiments>
The present invention has been described based on the above embodiment, but the present invention is not limited to the above embodiment, and various modifications can be easily made without departing from the spirit of the present invention. It can be guessed. In each embodiment, a part or a plurality of parts of the configuration of the other embodiments are added to the embodiment or replaced with a part or a plurality of parts of the configuration of the embodiment. You may make it comprise and modify. Hereinafter, an embodiment as a modification of the above embodiment will be described.

  The timing for transmitting the screen ID from the portable terminal 40 to the MFP 10 is not limited to the time when coordinate information or the like is transmitted in accordance with the pressing event (S24). For example, the screen ID may be transmitted from the portable terminal 40 to the MFP 10 at regular intervals.

  In FIG. 2, when the screen itself is changed, the screen itself is changed in view of the fact that the object indicating the location where the indication point should be located in instructing the processing executed by the MFP 10 is displaced in the display area. At this time, the input operation to the touch panel 46 of the portable terminal 40 is not accepted, but the present invention is not limited to this. For example, when the entire image displayed in the display area 20a of the LCD 20 of the MFP 10 is not changed, but only the object indicating the location where the instruction point should be located in instructing the processing executed by the MFP 10, the object is displaced. An input operation on the touch panel 46 of the mobile terminal 40 may not be accepted.

  A setting for sending LCD data and additional information corresponding to the LCD data from the MFP 10 by requesting the LCD data is not limited to the request for the LCD data and the request for the additional information separately from the portable terminal 40 to the MFP 10. It may be.

  In the processing of S46 and S54 in FIG. 4, the input operation to the touch panel 46 of the portable terminal 40 is not accepted during the animation display period on the MFP 10 side, but this is not restrictive. For example, the display of the display area 44a of the LCD 44 is changed according to the input operation, and the fact that the input operation is invalidated is further displayed on the display area 44a of the LCD 44, and the input event and coordinate information related to the input operation are displayed on the MFP 10 Transmission may not be performed.

  In the process of FIG. 4, the input operation to the touch panel 46 is not accepted during the period when the waiting process of S <b> 48 is performed. However, even during this period, the pinch-out operation for instructing the enlargement of the screen is performed. As described above, an input operation unrelated to the instruction to the MFP 10 may be accepted.

  In the method of FIG. 7B, the length of each coordinate component from the reference point to the predetermined point is set with the reference point as the drawing start coordinate (X, Y) and the predetermined point as the drawing end coordinate (Xe, Ye). The present invention is not limited to the information used as standardized information for specifying the position of the pointing point when the image has a prescribed size. For example, the predetermined point may be set by the coordinate component of the midpoint in the X-axis direction of the image and the coordinate component of the midpoint in the Y-axis direction. Even in this case, the length of each coordinate component from the reference point to the predetermined point can be used as normalization information for specifying the position of the indication point when the image has a specified size.

  The coordinate information transmitted from the portable terminal 40 to the MFP 10 includes standardized information for specifying the position of the indication point when the image has a specified size regardless of whether the image is enlarged or reduced (see FIG. 7 (a) enlargement factor k and (Xe-Y, Ye-Y) in FIG. For example, instead of FIG. 7A, coordinate components (AX, BY) having the drawing start coordinates as the origin and the enlargement factor k as the normalized information may be transmitted to the MFP 10 separately. Good. Even in this case, the MFP 10 can specify the position of the indication point when the image has a specified size. Further, information transmitted from the portable terminal 40 to the MFP 10 is not limited to information including standardized information. That is, if enlargement / reduction processing is not performed on the portable terminal 40, the MFP 10 can specify the position of the indication point when the image has a specified size without including the processing.

  The means for requesting the LCD data from the MFP 10 is not limited to that using a lifting operation on the touch panel 46 as a trigger. For example, a pressing operation on the touch panel 46 may be used as a trigger.

  The method for requesting LCD data from the MFP 10 is not limited to the method using an input event to the touch panel 46 as a trigger. For example, the LCD data may be requested at a predetermined time period. At this time, the screen ID is further transmitted from the portable terminal 40 to the MFP 10, and the MFP 10 displays the screen ID of the image displayed on the display area 44 a of the LCD 44 of the portable terminal 40 and the image displayed on the display area 20 a of the LCD 20 of the MFP 10. The LCD data may be transferred to the portable terminal 40 on condition that the screen ID does not match.

  The MFP 10 is not limited to the one provided with the LCD 20 and the touch panel 22. Even if these are not provided, if image data representing an image for guiding an input operation on the touch panel 46 is transmitted to the mobile terminal 40, the MFP 10 is operated in accordance with the input operation on the touch panel 46 of the mobile terminal 40. be able to. In this case, the MFP 10 cannot set an image displayed in the display area 20a of the LCD 20, but if an image for guiding an operation that can be accepted in the MFP 10 is set, the screen ID is checked. It is also possible to execute the processing of S128 to S140 in FIG. In this case, as a situation where it is determined that the screen IDs do not match, for example, when the mobile terminal 40, the MFP 10, or communication between the mobile terminal 40 and the MFP 10 is mixed with noise, or the mobile terminal 40 is temporarily There may be a case where a malfunction occurs due to a power shortage.

  The input unit (terminal side input unit, device side input unit) is not limited to the touch panels 22 and 46. For example, a device that detects pressing, lifting, dragging, and the like on the LCDs 20 and 44 by image recognition may be used.

  A communication method between the portable terminal 40 and the MFP 10 when operating the MFP 10 with the portable terminal 40 is not limited to Wi-Fi communication. For example, communication using the telephone network communication unit 50 may be used.

  The operation target (processing device) using the portable terminal 40 is not limited to the MFP 10. For example, you may operate the energy management apparatus of the house with which the house was equipped. In this case, if the telephone network communication unit 50 is used, the user activates an air conditioner or the like prior to returning home from a land far away from the house, or uses the power generated by the solar panel in preference to boiling water. It is also possible to give instructions. At this time, when the energy management apparatus itself includes a touch panel, it is possible to issue an instruction from the destination by an operation according to the touch panel operation in the house.

  The communication terminal is not limited to a terminal having a function other than the function related to the operation of the processing apparatus such as the MFP 10 such as a function of a telephone call via a telephone network, and is a terminal specialized for the operation of the processing apparatus. May be.

  In the above embodiment, part or all of the processing (software processing) realized by the CPU 52 executing the operation support program may be changed to hardware processing by an ASIC or the like.

  The “process executed by the processing device” in claim 1 includes a process for executing each function such as a process for executing a copy function, a process for executing a facsimile function, a process for executing a scanner function, and the like. Replacement processing is also applicable. For example, when the Fax icon is selected on the screen of FIG. 2A, a process of switching to the screen (image) of FIG.

  10 ... MFP, 40 ... mobile terminal.

Claims (14)

A display unit having a display area; an input unit that detects a point on the display area that is a point designated by the input medium; a communication unit that communicates with the processing device; and a control unit. In an operation support program for operating the processing device based on an input to the input unit and executed by a communication terminal comprising:
Transmission request means for requesting the processing device to transmit image data representing the image with respect to an image for guiding an input operation by the input medium for instructing processing executed by the processing device;
Image receiving means for receiving the image data transmitted in response to a request by the transmission requesting means;
Display control means for displaying an image represented by the received image data in the display area;
Image replacement notification receiving means for receiving that the processing device is gradually replacing the image;
Input information for transmitting the coordinate information of the indicated point on the display area detected by the input unit in a state where the image is displayed by the display control means, and the type of the input event for the input unit to the processing device Transmitting means to execute the control unit ,
While the input information transmitting means receives that the image is being replaced by the image replacement notification receiving means, while not transmitting the type of the input event to the processing device,
The transmission request means, while receiving the effect is being swapped image by middle notification receiving means replacing the image, the operation support program characterized that you request transmission of the image data.   The operation support program according to claim 1, wherein the transmission request unit requests transmission of the image data on condition that the transmission process of the input information transmission unit is executed. The types of input events transmitted by the input information transmitting means include selection of a position on the image displayed in the display area, and confirmation of the selection,
The operation according to claim 2, wherein the transmission request unit requests transmission of the image data on condition that an input event indicating that the selection is confirmed is transmitted by the input information transmission unit. Support program. The display unit is a terminal side display unit,
The input unit is a terminal side input unit,
The processing device includes: a device-side display unit having a display area; and a device-side input unit that detects a point on the display area of the device-side display unit and designated by an input medium; With
The image represented by the image data is displayed in a display area of the device-side display unit,
When changing the image displayed in the display area of the device-side display unit, the processing device has a function of gradually replacing the image before the change to the image after the change,
The transmission request unit waits for a predetermined period before requesting transmission of the image data when an input event indicating that the selection is confirmed is transmitted by the input information transmission unit. 3. The operation support program according to 3. 5. The operation support program according to claim 4, wherein the input information transmission unit does not transmit the type of an input event that occurs while the transmission request unit waits for a request for transmission of the image data. . The display unit is a terminal side display unit,
The input unit is a terminal side input unit,
The processing device includes: a device-side display unit having a display area; and a device-side input unit that detects a point on the display area of the device-side display unit and designated by an input medium; With
The image represented by the image data is displayed in a display area of the device-side display unit,
Causing the control unit to execute displacement notification receiving means for receiving from the processing device that the object in the image displayed in the display area of the device-side display unit is displaced;
The transmission request means requests transmission of the image data on the condition that the displacement notification receiving means receives the displacement when the image receiving means receives the image data. The operation support program according to claim 1 or 2. The input information transmission means, triggered by the input operation on the input unit, the feed and be one that performs the signal processing, the condition that the displacement notice receiving means receives the effect including the displacement, the processing The operation support program according to claim 6, wherein the transmission process to the apparatus is not executed.   The input information transmitting means is an object in an image displayed in the display area of the terminal-side display unit, and indicates a position where the instruction point should be in order to instruct a process to be executed by the processing device The operation support program according to claim 7, wherein when the object is displaced, the transmission process is not executed. When the input information transmitting means displays an animation indicating the degree of progress of processing executed by the processing device as a moving object in an image displayed in the display region of the device-side display unit, in the display region, The operation support program according to claim 7 or 8, wherein the transmission processing to the processing device can be executed.   The displacement notification receiving means for displaying the fact in the display area of the terminal-side display unit is executed by the control unit on condition that the displacement notification receiving unit receives that the displacement is included. The operation support program according to any one of 6 to 9. A display unit having a display area; an input unit that detects a point on the display area that is a point designated by the input medium; a communication unit that communicates with the processing device; and a control unit. A communication terminal comprising:
Each means to be executed by the control unit by the operation support program according to any one of claims 1 to 10 is provided as means realized by at least one of software processing and hardware processing. A characteristic communication terminal. The communication terminal comprising: a display unit having a display region; an input unit that detects a point on the display region that is a point designated by the input medium; a communication unit; and a control unit. In the processing device operated based on the input to the input unit,
Request receiving means for receiving, from the communication terminal, a transmission request for image data representing the image with respect to an image that guides an input operation using the input medium for instructing processing executed by the processing device;
Image transmission means for transmitting the image data to the communication terminal on the condition that the request reception means receives the transmission request;
Input information receiving means for receiving from the communication terminal the coordinate information of the indicated point on the display area detected by the input unit, and the type of input event for the input unit;
Execution means for receiving the coordinate information received by the input information receiving means and the type of the input event as input, and executing processing according to the coordinate information and the input event ;
When changing the image, it has a function of gradually replacing the image before the change to the image after the change,
If the image is being replaced gradually, an image replacement notification transmitting means for transmitting the fact to the communication terminal;
A processing apparatus comprising: The display unit is a terminal side display unit,
The input unit is a terminal side input unit,
A device-side display unit having a display area;
A device-side input unit for detecting a point on the display area of the device-side display unit and a point designated by the input medium;
The image displayed in the display area of the device-side display unit includes an image including the displacement of the object in the image,
The image transmission means transmits image data representing an image being displayed in a display area of the device-side display unit to the communication terminal on condition that the request reception means receives the transmission request. In addition, the displacement is completed even when the transmission request is received by the request reception means on the condition that the object in the image displayed in the display area of the device-side display unit is displaced. The processing apparatus according to claim 12, further comprising a waiting unit that waits until the image is transmitted, and transmitting image data representing an image at the completion to the communication terminal when the displacement is completed. The display unit is a terminal side display unit,
The input unit is a terminal side input unit,
A device-side display unit having a display area;
A device-side input unit for detecting a point on the display area of the device-side display unit and a point designated by the input medium;
The image displayed in the display area of the device-side display unit includes an image including the displacement of the object in the image,
The image transmission means transmits image data representing an image being displayed in a display area of the device-side display unit to the communication terminal on condition that the request reception means receives the transmission request. ,
The processing apparatus according to claim 12, further comprising a displacement notifying unit that notifies the communication terminal of the fact that an image being displayed in the display area of the apparatus-side display unit includes an animation.

Images