JP6390131B2 - Process execution system, process execution device, and process execution program - Google Patents

Description

The present invention relates to a process execution system, a process execution apparatus, and a process execution program.

  Japanese Patent Application Laid-Open No. 2004-228561 discloses a system that can remotely control an image forming apparatus from a portable device. In the system, the portable device displays the operation screen received from the image forming apparatus on the display unit, accepts an operation input by the user according to the operation screen, and transmits an instruction according to the operation to the image forming apparatus. The image forming apparatus displays a screen corresponding to the instruction received from the portable device.

JP 2012-70068 A

An object of the present invention is to provide a process execution system, a process execution apparatus, and a process execution program that can be remotely operated from a terminal using a method different from that of the above-mentioned Patent Document 1.

In order to achieve this object, a process execution system of the present invention includes a process execution device and a terminal. The terminal includes a voice input unit, a terminal communication unit that communicates with the process execution device, and terminal control. And the terminal control unit converts the voice into text data when the voice is input via the voice input unit, and the text data converted by the voice conversion unit. Is transmitted to the process execution device via the terminal communication unit, and the process execution device communicates with the terminal , a display unit on which a screen is displayed, a storage unit, and the terminal. And a control unit, wherein the screen includes an operation element associated with the process, and the storage unit stores text data associated with the operation element for each operation element, and the control Department is front Via the communication unit, a text determining unit text data to determine whether it is stored in the storage unit corresponding to the text data received from the terminal, by the text determining unit, the text data corresponding to text data the received Is determined to be stored in the storage unit, a process execution unit that executes a process according to the operator associated with the text data.

The process execution system of the present invention can be configured in various modes such as a process execution apparatus, a control apparatus that controls the process execution apparatus, a process execution method, a program that realizes the process execution system, and a recording medium that records the program. . Further, a terminal that can be used in the processing execution system of the present invention can be configured in various modes such as a control device that controls the terminal and a program that realizes the terminal.

According to the process execution system of the first aspect , when the voice is input through the voice input unit , the terminal converts the voice into text data and transmits the text data to the process execution device. In the processing execution device, for each operation element included in the screen, text data associated with the operation element is stored in the storage unit. When text data corresponding to the text data received from the terminal is stored in the storage unit, processing corresponding to the operator associated with the text data is executed. Therefore, the processing execution apparatus can be remotely operated according to text data corresponding to the operation element received from the terminal .

According to the processing execution system of the second aspect, in addition to the effect of the first aspect, even if the operator is associated with the attribute for hiding the text, the text data corresponding to the operator from the terminal Remote operation can be performed in response to reception.

According to the processing execution system of claim 3, in addition to the effect of claim 2, remote operation of the screen on which the operation element is displayed on the background image is performed in response to reception of text data corresponding to the operation element from the terminal. can do.

According to the process execution system of the fourth aspect, in addition to the effect produced by any one of the first to third aspects, the following effect is obtained. If the screen being displayed is a screen for entering text, the received text data is input to the screen being displayed without executing processing based on the text data received from the terminal . Accept as text. Therefore, it is possible to perform remote operation using text data from the terminal, and it is possible to input text on a screen for inputting text.

According to the processing execution system of claim 5, in addition to the effect of claim 4, the following effect is obtained. Even if the screen being displayed is a screen for inputting text, the text data received from the terminal is composed of character types that can be input to the screen being displayed. Acceptance is done. Therefore, it is possible to prevent problems associated with the input of a character type that cannot be input to the screen for inputting text that is being displayed.

According to the processing execution system of the sixth aspect, in addition to the effect produced by the fourth or fifth aspect, the following effect is obtained. Even if the screen being displayed is a screen for inputting text, if text data corresponding to the text data received from the terminal is stored in the storage unit, the operation associated with the received text data Processing according to the child is executed. Therefore, the operator included in the screen for inputting text on the screen being displayed can be remotely operated from the terminal .

According to the processing execution system of the seventh aspect, in addition to the effect produced by any one of the first to sixth aspects, the following effect is produced. Received from the terminal together with the text data, specifying information for specifying a screen displayed on the terminal, in the case does not specify a screen displayed on the display unit, the process based on the text data received along with the identification information Execution is not performed. Therefore, it is possible to suppress a malfunction caused by a mismatch between the screen that is the target of remote operation at the terminal and the screen that is being displayed on the display unit. In that case, since the second error information indicating that the screen being displayed on the display unit is different from the screen displayed on the terminal is output, it is possible to inform the terminal that there is a screen flaw. Can do.

According to the processing execution system of the eighth aspect, in addition to the effect produced by any one of the first to seventh aspects, the following effect is produced. When text data corresponding to text data received from the terminal is not stored in the storage unit, third error information indicating that there is no operator corresponding to the text data is output to the terminal . Therefore, this can be reported to the terminal .

According to the process execution system of the ninth aspect, in addition to the effect produced by any one of the first to eighth aspects, the following effect is obtained. If text data corresponding to the text data received from the terminal is not stored in the storage unit, uppercase and lowercase letters and / or one of the received text data or text data stored in the storage unit Alternatively, the combination of full-width and half-width is changed, and it is determined whether the changed text data corresponds to the other text data. And also when the said determination is affirmed, the text data equivalent to the text data received from the terminal is memorize | stored in the memory | storage part, and the process based on the received text data is performed. Therefore, even when the terminal transmits text data having a combination of uppercase and lowercase letters and / or a combination of full-width and half-width, the corresponding process can be executed.

According to the processing execution system of the tenth aspect, in addition to the effect produced by any one of the first to ninth aspects, the following effect is produced. While an animation display whose contents change with the passage of time is being performed, there is a possibility that a discrepancy may occur between the screen displayed on the display unit and the screen targeted for remote operation on the terminal . On the other hand, since the text data received from the terminal is invalidated while the animation display is displayed on the display unit, it is possible to suppress a malfunction according to the mismatch between the two screens.

According to the processing execution device of the eleventh aspect, for each operator included in the screen, text data associated with the operator is stored in the storage unit. When text data corresponding to the text data received from the terminal is stored in the storage unit, processing corresponding to the operator associated with the text data is executed. In addition, when the displayed screen is a screen for inputting text, the received text data is input to the displayed screen without performing processing based on the text data received from the terminal. As received text. Therefore, it is possible to perform remote operation using text data from the terminal, and it is possible to input text on a screen for inputting text.
According to the process execution program according to claim 1 wherein the same advantages as the processing execution apparatus according to claim 1 1, wherein.

It is a block diagram which shows the electrical structure of a multifunction peripheral device and a terminal. It is a figure explaining the outline of a structure of the screen displayed on LCD. It is a flowchart which shows the terminal side process at the time of remote operation. 6 is a flowchart illustrating MFP-side processing during remote operation. It is a flowchart which shows a text process. It is a flowchart which shows a collation process.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a block diagram showing an electrical configuration of a multifunction peripheral device (hereinafter referred to as “MFP”) 10 and a terminal 200 that can communicate with the MFP 10. The MFP 10 is an embodiment of the processing execution apparatus of the present invention.

  The MFP 10 has various functions such as a print function, a scan function, a copy function, and a facsimile function. The MFP 10 is configured to be able to execute processing related to various functions by remote operation from a terminal 200 on which an operation support application (hereinafter referred to as “operation support application”) 212b is mounted. In particular, the MFP 10 is configured to be able to execute processing according to text data received from the terminal 200.

  The MFP 10 is mainly provided with a CPU 11, flash memory 12, RAM 13, operation keys 15, LCD 16, touch panel 17, scanner unit 18, printer unit 19, NCU 20, modem 21, and wireless communication unit 23. These units are connected to each other via the input / output port 24. The CPU 11 controls various functions of the MFP 10 according to fixed values and programs stored in the flash memory 12, data stored in the RAM 13, or various signals transmitted / received via the NCU 20, and input / output Each unit connected to the port 24 is controlled.

  The flash memory 12 is a nonvolatile memory, and stores a control program 12a for controlling the operation of the MFP 10 and the like. Each process shown in the flowcharts of FIGS. 4 to 6 described later is a process executed by the CPU 11 in accordance with the control program 12a. Further, the flash memory 12 is prepared with image data of each screen image that can be displayed on the LCD 16 for the entire displayable screen. Although details will be described later, the image data of each screen is composed of image data of the background of the screen and data of various components arranged on the background. The component is composed of an image, a character string, or a combination of an image and a character string. Therefore, the component data is either image data of an image constituting the component, character string data (hereinafter referred to as “text data”), or a combination of image data and text data.

  The flash memory 12 is provided with an image data map (not shown) for managing the image data of each screen image. In the image data map, the screen ID is associated with the image data of each screen. “Screen ID” is information specifying each screen.

  The RAM 13 is a rewritable volatile memory having a temporary area for temporarily storing various data when the CPU 11 executes the control program 12a.

  The operation key 15 is a mechanical key provided on the casing of the MFP 10 and accepts various setting values and instructions from the user. The LCD 16 is a liquid crystal display device and displays various screens. A touch panel 17 is provided on the LCD 16 so as to overlap the LCD 16. For example, when an indicator such as a finger or a stick touches or approaches the screen of the LCD 16, the touch panel 17 detects the touched or approached position and inputs the detected position to the MFP 10. The scanner unit 18 reads a document and converts it into image data. The printer unit 19 prints an image based on the image data on a recording sheet. The modem 21 modulates image data to be transmitted at the time of FAX transmission into a signal that can be transmitted to a telephone line network (not shown) and transmits the signal via the NCU 20 or is input from the telephone line network via the NCU 20. The received signal is demodulated into image data. The NCU 20 connects a telephone line network (not shown) and the MFP 10, and controls the connection state with the telephone line network by closing or disconnecting the line according to an instruction from the modem 21.

  The wireless communication unit 23 is an interface for performing wireless communication using a wireless LAN. In the present embodiment, the communication performed by the wireless communication unit 23 is wireless communication based on a wireless LAN compliant with the IEEE802.11b / g / n standard. In the present embodiment, the wireless communication unit 23 is based on the Wi-Fi (registered trademark) standard with the terminal 200 in an infrastructure mode via an access point (hereinafter referred to as “AP”) 50 that is a relay device. The connection is established by wireless communication (hereinafter referred to as “Wi-Fi communication”). In the following description, communication performed between the MFP 10 and the terminal 200 is Wi-Fi communication via the AP 50 unless otherwise specified.

  The terminal 200 of this embodiment is a tablet terminal. The terminal 200 is mainly provided with a CPU 211, a flash memory 212, a RAM 213, an operation key 215, an LCD 216, a touch panel 217, a memory card interface (memory card I / F) 218, a voice input unit 219, and a wireless communication unit 223. These units are connected to each other via an input / output port 224. The CPU 211 controls each unit connected to the input / output port 224 according to a fixed value, a program, or the like stored in the ROM 212 or the like.

  The flash memory 212 is a rewritable nonvolatile memory. The flash memory 212 stores an operating system 212a and an operation support application 212b. Hereinafter, the “operating system” may be referred to as “OS”. In the following description, the CPU 211 that executes a program such as an application or an operating system may be simply described by a program name. For example, the description “application” may mean “CPU 211 executing the application”. The OS 212a is basic software for realizing the standard functions of the terminal 200. In this embodiment, an Android (registered trademark) OS is adopted as an example of the OS 212a.

  The operation support application 212b is an application program for remotely operating a device such as the MFP 10. The operation support application 212b is installed in the terminal 200 by the user, and enables remote operation of the corresponding device from the terminal 200. Each process shown in the flowchart of FIG. 3 to be described later is a process executed by the CPU 211 in accordance with the operation support application 212b.

  The operation key 215 is a mechanical key provided on the casing of the terminal 200, and accepts various setting values and instructions from the user. LCD 216, touch panel 217, and wireless communication unit 223 are the same as LCD 16, touch panel 17, and wireless communication unit 23 described above, respectively. The memory card I / F 218 is an interface to which a rewritable nonvolatile memory card (not shown) is attached, and controls writing or reading of data with respect to the memory card. The voice input unit 219 is a voice input device such as a built-in microphone.

  With reference to FIG. 2, the outline of the structure of the screen displayed on LCD16 is demonstrated. FIG. 2A shows a Home screen 80 which is an example of a screen displayed on the LCD 16. The Home screen 80 includes a background 81 and various parts arranged on the background 81. Examples of components arranged on the background 81 include an image such as an image 82, buttons such as buttons 83 and 84, and a label such as a label 85.

  In the example shown in FIG. 2A, the button 83 is a button for instructing use of the facsimile function. The button 83 includes an image 83a reminiscent of the facsimile function, and a character string 83b described as “Fax” arranged below the image 83a. On the other hand, the button 84 is a button for instructing to switch the screen displayed on the LCD 16 from the Home screen 80 to a screen adjacent to the right side of the screen 80. The button 84 is composed of a triangular image 84 in which one of the vertices faces the right side. A character string like the button 83 is not displayed on the button 84.

  The MFP 10 manages how the backgrounds and parts constituting each screen overlap by using a tree structure. For example, FIG. 2B shows a tree structure of the Home screen 80 shown in FIG. The highest level of the tree structure is the background 81, and the lower the level, the more components are arranged on the near side as viewed from the user. The button 83 is positioned below the image 82. That is, the button 83 is arranged on the screen 82, that is, on the near side as viewed from the user.

  As described above, the button 83 includes the image 83a and the character string 83b. Display attributes are associated with the character string 83b. The character string associated with the display attribute is displayed on the screen. Therefore, the character string 83b, that is, the character string “Fax” is displayed on the Home screen 80 as shown in FIG.

  On the other hand, the button 84 is positioned below the background 81. That is, the button 84 is disposed on the background 81. Similar to the button 83, the button 84 also includes an image 84a and a character string 84b. However, unlike the character string 83 b of the button 83, a non-display attribute is associated with the character string 84 b of the button 84. A character string associated with a non-display attribute is not displayed on the screen. Therefore, the character string 84b, that is, the character string “Right” is not displayed on the Home screen 80 as shown in FIG.

  Although details will be described later, in the present embodiment, various buttons included in the screen such as the button 83 and the button button 84 can be operated according to the text data received from the terminal 200. That is, the MFP 10 according to the present embodiment can be remotely operated from the terminal 200 even if the button 84 does not display a character string, such as the button 84.

  FIG. 3 is a flowchart showing terminal-side processing at the time of remote operation executed by the CPU 211 of the terminal 200. This process is a process for the terminal 200 to remotely operate the MFP 10. This process is started on the condition that the preliminary process for performing a remote operation between the terminal 200 and the MFP 10 is completed.

  Prior to the description of FIG. 3, the pre-processing will be described. The pre-processing starts when the user operates the MFP 10 and sets the remote operation to be valid. Next, the user operates the terminal 200 to activate the operation support application 212b, and causes the terminal 200 on which the operation support application 212b is activated to search for the MFP 10 in which the remote operation is enabled. The terminal 200 registers the searched MFP 10 as a remote operation target in the terminal 200, and transmits registration information indicating registration, the IP address of the terminal 200, and the like to the registered MFP 10 by Wi-Fi communication. Receiving these, the MFP 10 establishes a communication link with the terminal 200. The pre-processing is completed when the communication link is established. As a result, the terminal 200 is in a state where the MFP 10 can be remotely operated.

  The CPU 211 acquires image data corresponding to the screen being displayed on the LCD 16 from the MFP 10 (S301). Hereinafter, the image data of the screen that is acquired (or received) from the MFP 10 and displayed on the LCD 16 will be referred to as “LCD data”. Specifically, the CPU 211 first requests LCD data from the MFP 10. The CPU 211 acquires LCD data returned from the MFP 10 according to the request. In the present embodiment, the LCD data is bitmap data.

  The CPU 211 acquires additional information from the MFP 10 (S302). The additional information includes a screen ID corresponding to the screen being displayed on the LCD 16. The CPU 211 stores the acquired additional information in the RAM 213. The CPU 211 displays an image based on the LCD data acquired in S301, that is, an image corresponding to the screen being displayed on the LCD 16 on the LCD 216 of the terminal 200 (S303). Which screen the image displayed on the LCD 216 corresponds to among the screens that can be displayed on the LCD 16 of the MFP 10 can be specified according to the screen ID stored in the RAM 213.

  The CPU 211 waits for an event to occur (S304). When an audio event occurs (S305: Yes), the CPU 211 executes the processes of S306 to S314 described later. Note that the “voice event” is an event that occurs in response to a voice input via the voice input unit 219.

  When a voice event occurs (S305: Yes), the CPU 211 receives the voice event, and converts a voice (hereinafter referred to as “input voice”) input via the voice input unit 219 into a text character string. (S306). Since the process of converting speech into a text character string is a well-known technique, details thereof are omitted, but when there are a plurality of converted text character string candidates, a representative one of them is included. A string is selected. In the present embodiment, it is assumed that the corresponding language for conversion is English.

  The CPU 211 transmits a text instruction event, text data, and a screen ID to the MFP 10 (S307). The text data transmitted in S307 is text character string data obtained by the conversion in S306. Further, the screen ID transmitted in S307 is a screen ID corresponding to the image displayed on the LCD 216 at the timing when the voice is input.

  After the process of S307, the CPU 211 determines whether an error notification indicating a mismatch of screen IDs has been received from the MFP 10 (S308). Although details will be described later with reference to FIG. 5, the MFP 10 determines whether the screen ID transmitted together with the text instruction event in S <b> 307 matches the screen ID of the screen currently displayed on the LCD 16. The terminal 200 is notified of an error notification indicating that. If the terminal 200 receives the error notification within a predetermined time (for example, 5 seconds) after the processing of S307, the CPU 211 affirms the determination of S308. In other cases, the CPU 211 denies the determination of S308.

  If the CPU 211 affirms the determination in S308, that is, if the screen ID transmitted together with the text instruction event in S307 does not match the screen ID of the screen currently displayed on the LCD 16 (S308: Yes), the CPU 211 Similar to S301 to S303 described above, the processes of S309 to S311 are executed, and the process proceeds to S304. Thereby, an image corresponding to the screen being displayed on the LCD 16 can be displayed on the LCD 216.

  On the other hand, if the CPU 211 denies the determination in S308 (S308: No), the CPU 211 determines whether an error notification indicating that the corresponding text does not exist is notified from the MFP 10 (S312). Although details will be described later with reference to FIG. 5, the MFP 10 matches the text data transmitted together with the text instruction event in S <b> 307 in the character string associated with the button included in the screen being displayed on the LCD 16. If there is no character string to be determined, an error notification to that effect is sent to the terminal 200. If the terminal 200 receives the error notification within a predetermined time (for example, 10 seconds) after the processing of S307, the CPU 211 affirms the determination of S312. In other cases, the CPU 211 denies the determination of S312.

  When the CPU 211 affirms the determination in S312 (S312: Yes), the CPU 211 displays an error display corresponding to the content of the error notification on the LCD 216 (S314), and the process proceeds to S304. The error display displayed in S314 is displayed as a pop-up display that is automatically deleted after a predetermined time (for example, 5 seconds), for example.

  On the other hand, when the CPU 211 denies the determination in S312, the CPU 211 determines whether an error notification indicating that text input is impossible is received from the MFP 10 (S313). Although details will be described later with reference to FIG. 5, when the screen for inputting text to the MFP 10 (hereinafter referred to as “text input screen”) is displayed on the LCD 16, the MFP 10 performs a text instruction event in S 307. In addition, it is determined whether the text data transmitted together with the text data is composed of character types that can be input to the displayed screen. When the MFP 10 denies the determination, the MFP 10 notifies the terminal 200 of an error notification to that effect. If the terminal 200 receives the error notification within a predetermined time (for example, 10 seconds) after the processing of S307, the CPU 211 affirms the determination of S313. In other cases, the CPU 211 denies the determination of S313.

  When the CPU 211 affirms the determination in S313 (S313: Yes), the CPU 211 executes the process in S314. On the other hand, when the CPU 211 denies the determination of S313 (S313: No), the CPU 211 proceeds to S304.

  When receiving an event other than the audio event (S305: No), the CPU 211 executes a process according to the received event as another process (S315), and shifts the process to S304. The event to be processed in S315 is, for example, various events that are generated in response to various operations such as a pressing operation, a drag operation, and a lifting operation performed on the touch panel 217. Since the process of S315 is a process unrelated to the present invention, a specific description thereof is omitted.

  FIG. 4 is a flowchart showing MFP-side processing during remote operation, which is executed by the CPU 11 of the MFP 10. This process is a process for operating the MFP 10 in response to a remote operation from the terminal 200. This process is started on the condition that the above-described preliminary process is completed, as in FIG.

  The CPU 11 waits for reception of an event (S401). When the CPU 11 receives the LCD data request event (S402: Yes), the CPU 11 executes processing of S403 and S404 described later. The “LCD data request event” is an event that occurs when a request for LCD data is received from the terminal 200.

  When the LCD data request event is received (S402: Yes), the CPU 11 refers to the image data map provided in the flash memory 12 and acquires the ID corresponding to the screen being displayed on the LCD 16 (S403). The CPU 11 transmits LCD data corresponding to the screen being displayed on the LCD 16 and additional information to the terminal 200 (S404), and the process proceeds to S401. The additional information transmitted to the terminal 200 in S404 includes the screen ID acquired in S403.

  When the CPU 11 receives a main body screen transition event (S402: No, S405: Yes), the CPU 11 executes a process of transitioning the screen of the LCD 16 according to the received event (S406), and shifts the process to S401. The “main body screen transition event” is an event that occurs when a screen transition of the LCD 16 is required. The transition of the screen of the LCD 16 includes a screen transition accompanying a remote operation by the terminal 200 and a screen transition that occurs regardless of the remote operation by the terminal 200. The latter corresponds to, for example, a display that informs the user of a situation that should be notified to the user in the MFP 10, for example, a display that is displayed on the LCD 16 as a pop-up display. In the present embodiment, in the image data map, different screen IDs are associated with a screen including a display in response to occurrence of a situation to be notified to the user and a screen not including the display. .

  When the CPU 11 receives the text instruction event (S405: No, S407: Yes), the CPU 11 executes the processes of S408 and S409 described later. The “text instruction event” is an event that occurs when a text instruction event is received from the terminal 200.

  When the text instruction event is received (S405: No, S407: Yes), the CPU 11 determines whether the animation display of the screen transition is being executed (S408). The “screen transition animation display” is a display that gradually changes from a screen displayed on the LCD 16 to another screen at a predetermined speed.

  When the CPU 11 determines that the animation display of the screen transition is not being executed (S408: No), the CPU 11 executes text processing (S409), and shifts the processing to S401. Text processing (S409) is processing according to the text data received from the terminal 200 together with the text instruction event. Details of the text processing (S409) will be described later with reference to FIG. 5. However, when the received text data matches a character string associated with a button included in the screen being displayed on the LCD 16, Treated as a button operation.

  On the other hand, when the CPU 11 determines in S408 that the screen transition animation display is being executed (S408: Yes), the CPU 11 proceeds to S401. Therefore, even when text data is received from the terminal 200 during execution of the screen transition animation display, the text data is invalidated.

  While the screen transition animation is being displayed, if the image displayed on the LCD 216 at the timing of voice input does not match the screen being displayed on the LCD 16, the processing intended at the timing of voice input Different processing may be performed. Accordingly, since the text data received from the terminal 200 is invalidated while the screen transition animation is displayed, it is possible to prevent the MFP 10 from malfunctioning due to the screen mismatch.

  When the CPU 11 receives an event other than the LCD data request event, the main body screen transition event, or the text instruction event (S407: No), the CPU 11 executes processing according to the received event as other processing (S410). The process proceeds to S401. Examples of events to be processed in S410 include various events that occur in response to various operations such as pressing operations performed on the terminal 200. Since the process of S410 is a process not related to the present invention, a specific description thereof will be omitted.

  FIG. 5 is a flowchart showing the text processing (S409) described above. The CPU 11 acquires the screen ID and text data received from the terminal 200 together with the text instruction event (S501, S502). Hereinafter, the text data acquired in S502 is referred to as “acquired text”.

  The CPU 11 determines whether the screen ID acquired in S501 matches the screen ID corresponding to the screen currently displayed on the LCD 16 (S503). When the CPU 11 determines that the two screen IDs match (S503: Yes), the CPU 11 executes a process of collating the acquired text with the character string included in the screen being displayed on the LCD 16 (S504). Note that the “character string included in the screen being displayed on the LCD 16” is associated with a non-display attribute, and a character string that is not displayed on the screen, for example, the button 84 displayed on the Home screen 80 (FIG. 2). The associated character string 84a “Right” is also included. Hereinafter, the “character string included in the screen being displayed on the LCD 16” may be referred to as “screen text”.

  Details of the collation processing (S504) will be described later with reference to FIG. The CPU 11 determines whether there is a screen text that matches the acquired text as a result of the collation process (S504) (S505). When the CPU 11 determines that there is screen text that matches the acquired text (S505: Yes), the CPU 11 determines whether the screen text that matches the acquired text is a character string associated with a button in the screen. (S506).

  Specifically, when the screen text that matches the acquired text is, for example, a character string 83a (Fax) associated with the button 83 or a character string 84a (Right) associated with the button 84, the CPU 11 The determination in S506 is affirmed. On the other hand, if the screen text that matches the acquired text is, for example, a character string (Home) associated with the label 85 that is not a button, the CPU 11 denies the determination in S506.

  When the CPU 11 affirms the determination of S506 (S506: Yes), the CPU 11 executes a pressing / leaving process for the button corresponding to the corresponding screen text, that is, the button associated with the character string matching the acquired text. (S507), and this process ends. The pressing / lifting process (S507) is the same process as when a pressing operation and a subsequent lifting operation are performed on the corresponding button via the touch panel 17. Note that when the screen of the LCD 16 changes due to the execution of the pressing / leaving process (S507), a main body screen transition event occurs.

  According to the processing of S505 to S507, when there is a button associated with a character string that matches the text data received from the terminal 200 (that is, the acquired text), a pressing operation is performed on the button. Thereafter, the same processing is executed when a lifting operation is performed at that position. Therefore, the MFP 10 can be remotely operated in accordance with the text data output from the terminal 200. In particular, the screen text that is the target of matching with the acquired text contains a character string that is not displayed on the screen, that is, a character string that is associated with a non-display attribute, so the character string is not displayed on the screen. Buttons can also be targeted for remote operation from the terminal 200.

  On the other hand, if the CPU 11 determines in S503 that the acquired screen ID does not match the screen ID corresponding to the screen currently displayed on the LCD 16 (S503: No), the CPU 11 issues an error notification indicating a mismatch between the screen IDs. The terminal 200 is notified (S514), and this process ends. As a result of the process of S514, the CPU 211 of the terminal 200 affirms the process of S308 in FIG.

  In the present embodiment, when the screen ID acquired together with the text data from the terminal 200 and the screen ID corresponding to the screen being displayed on the LCD 16 do not match, the CPU 11 does not execute the processing after S504, and notifies the terminal of an error. 200 is notified. Therefore, it is possible to suppress execution of a process different from the process intended at the timing when voice input is performed.

  On the other hand, when the CPU 11 denies the determination in S506 (S506: No), the CPU 11 notifies the terminal 200 of an error notification indicating that the corresponding text does not exist (S508), and ends this process. As a result of the processing of S508, the CPU 211 of the terminal 200 affirms the processing of S312 in FIG. 3 and displays an error display on the LCD 216. Therefore, it can be reported to the user who performs remote operation using the terminal 200 that there is no button corresponding to the input voice.

  When the CPU 11 determines that there is no screen text that matches the acquired text (S505: No), the CPU 11 determines whether the screen being displayed on the LCD 16 is a text input screen (S509). The “text input screen” is a screen for inputting text to the MFP 10, for example, a screen including a soft keyboard for inputting text. When the CPU 11 determines that the screen being displayed on the LCD 16 is not a text input screen (S509: No), the CPU 11 proceeds to S508 and notifies the terminal 200 of an error notification indicating that the corresponding text does not exist. .

  On the other hand, when the CPU 11 determines that the screen displayed on the LCD 16 is a text input screen (S509: Yes), the CPU 11 determines whether the acquired text can be input to the displayed text input screen. (S510). Specifically, when the acquired text is composed of character types that can be input to the displayed text input screen, the CPU 11 affirms the determination in S510. When the CPU 11 affirms the determination of S510 (S510: Yes), the CPU 11 accepts the acquired text as the input text (S511), and displays the accepted input text in the input area (input field) on the text input screen (S512). ), This process is terminated.

  According to the processing of S505 and S509 to S512, when the screen being displayed on the LCD 16 is a text input screen, the text data received from the terminal 200 can be input to the text input screen as input text. In particular, since the input text is accepted on the condition that the text data received from the terminal 200 is composed of character types that can be input to the text input screen being displayed, there are character types that cannot be input to the text input screen. Problems associated with input can be prevented.

  Even when the screen being displayed is a text input screen, if the CPU 11 receives text data that matches the character string associated with the button included in the text input screen from the terminal 200, the CPU 11 performs S507. Execute the process. Therefore, the buttons included in the text input screen can be operated according to the text data received from the terminal 200. As a result, text can be input to the text input screen by remote operation from the terminal 200, and various buttons included in the text input screen can be operated. Various buttons included in the text input screen include, for example, a button for confirming the input text (for example, an OK button), a button for moving to the next page or the next item (for example, a NEXT button), or the like. .

  On the other hand, when the text input screen being displayed has character type restrictions, and the acquired text includes a character type that cannot be input to the text input screen being displayed, the CPU 11 denies the determination of S510. For example, when the text input screen is a screen for inputting an IP address, only numeric input is allowed on the screen, and alphabetic input is restricted. Therefore, when the acquired text includes English characters on the screen for inputting the IP address, the CPU 11 denies the determination in S510.

  When CPU11 denies determination of S510 (S510: No), CPU11 notifies the error notification which shows that a text input is impossible to the terminal 200 (S513), and complete | finishes this process. As a result of the processing of S513, the CPU 211 of the terminal 200 affirms the processing of S313 in FIG. 3 and displays an error display on the LCD 216. Therefore, it is possible to inform the user who performs remote operation using the terminal 200 that text input cannot be performed.

  FIG. 6 is a flowchart showing the above-described collation process (S504). The CPU 11 searches for a character string (that is, screen text) included in the screen being displayed on the LCD 16 (S601). As described above, the screen text to be searched includes not only a character string displayed on the screen but also a character string that is associated with a non-display attribute and is not displayed on the screen. The CPU 11 selects one screen text among the searched screen texts as a comparison target (S602).

  For the screen text and the acquired text selected as the comparison target, the CPU 11 acquires the first one character of both texts as a comparison character (S603). The CPU 11 compares the acquired two comparison characters (S604). When the CPU 11 determines that both comparison characters are the same character (S605: Yes), the CPU 11 determines whether the next character exists in the screen text to be compared and the acquired text (S606). When the CPU 11 determines that the next character exists in both texts (S606: Yes), the CPU 11 acquires the next character in both texts (S609), and the process proceeds to S604.

  When the CPU 11 determines that the next character does not exist in both texts (S606: No), the CPU 11 determines whether all characters have been processed for the screen text and the acquired text to be compared (S607). If the CPU 11 determines that all characters have been processed for both texts (S607: Yes), it indicates that the current screen text to be compared matches the acquired text. Therefore, in such a case, the CPU 11 determines that there is a screen text that matches the acquired text (S608), and ends this process.

  On the other hand, if the CPU 11 determines that all the characters have not been processed for both texts (S607: Yes), there is an unprocessed character in either the screen text or the acquired text, and the number of characters in both texts is different. Show. In such a case, the CPU 11 determines whether the next screen text exists (S619).

  If the CPU 11 determines that the next screen text exists (S619: Yes), the CPU 11 selects the next screen text as a comparison target (S620), and the process proceeds to S603. On the other hand, when the CPU 11 determines that the next screen text does not exist (S619: No), the CPU 11 determines that there is no screen text that matches the acquired text (S621), and ends this process.

  In S605, when the CPU 11 determines that both comparison characters are not the same character (S605: No), the CPU 11 executes uppercase / lowercase conversion on the comparison character of the target acquired text among the two comparison characters ( S610). Specifically, when the comparison character of the acquired text is uppercase, the CPU 11 converts the comparison character to lowercase. On the other hand, when the target comparison character is a lower case letter, the CPU 11 converts the comparison letter to an upper case letter.

  The CPU 11 compares the comparison character of the acquired text subjected to uppercase / lowercase conversion and the comparison character of the screen text (S611). As a result of the comparison in S611, when the CPU 11 determines that both comparison characters are the same character (S612: Yes), the CPU 11 shifts the processing to S606.

  On the other hand, if the CPU 11 determines that both comparison characters are not the same character (S612: No), the CPU 11 executes half-width / full-width conversion for the comparison character of the acquired text that has been converted to uppercase / lowercase in S610 (S613). ). Specifically, when the target comparison character is a half-width character, the CPU 11 converts the comparison character into a full-width character. On the other hand, if the target comparison character is a full-width character, the CPU 11 converts the comparison character into a half-width character.

  The CPU 11 compares the comparison character of the acquired text subjected to the half-width / full-width conversion with the comparison character of the screen text (S614). As a result of the comparison in S614, when the CPU 11 determines that both comparison characters are the same character (S615: Yes), the CPU 11 proceeds to S606. On the other hand, when the CPU 11 determines that both comparison characters are not the same character (S615: No), the CPU 11 executes uppercase / lowercase conversion on the comparison character of the acquired text subjected to the half-width / full-width conversion in S613 (S616). ).

  The CPU 11 compares the comparison character of the acquired text that has undergone uppercase / lowercase conversion in S616 with the comparison character of the screen text (S617). As a result of the comparison in S617, when the CPU 11 determines that both comparison characters are the same character (S618: Yes), the CPU 11 shifts the processing to S606. When the CPU 11 determines that both comparison characters are not the same character (S618: No), the CPU 11 shifts the process to S619.

  According to the processing of S610 to S618, even if the original acquired text does not match the original screen text, the combination of the uppercase and lowercase letters and / or full-width and half-width of the acquired text is changed and the comparison is performed again. Done. Therefore, even if the terminal 200 transmits text data having a combination of uppercase and lowercase letters and / or full-width and half-width different from the character string included in the screen being displayed on the LCD 16, the corresponding processing is performed. It can be executed properly.

  According to the MFP 10 of the present embodiment, the buttons included in the screen being displayed on the LCD 16 can be operated according to the text data received from the terminal 200. Thereby, the MFP 10 can be remotely operated from the terminal 200. On the other hand, when the screen being displayed on the LCD 16 is a text input screen, it is also possible to input text from the terminal 200 to the screen while remotely operating the screen from the terminal 200.

In the above embodiment, the MFP 10 is an example of a process execution device. The control program 12a is an example of a process execution program. The CPU 11 is an example of a control unit. The LCD 16 is an example of a display unit. The flash memory 12 is an example of a storage unit. The wireless communication unit 23 is an example of a communication unit. The Home screen 80 is an example of a screen. The buttons 83 and 84 are an example of an operation element. The background 81 is an example of a background image. CPU11 which performs the process of S504 is an example of a text determination means. The CPU 11 that executes the process of S507 is an example of a process execution unit. CPU11 which performs the process of S406 is an example of a display means. CPU11 which performs the process of S509 is an example of a 1st screen determination means. CPU11 which performs the process of S511 is an example of a text reception means. The CPU 11 that executes the process of S510 is an example of a character type determination unit. The CPU 11 that executes the process of S513 is an example of a first error output unit. The CPU 11 that executes the process of S503 is an example of a second screen determination unit. The CPU 11 that executes the process of S514 is an example of a second error output unit. The CPU 11 that executes the process of S508 is an example of a third error output unit. The CPU 11 that executes the processes of S604 and S605 is an example of a first determination unit. CPU11 which performs the process of S610-S618 is an example of a 2nd determination means. The CPU 11 that executes the process of S408 is an example of an invalidating unit. The terminal 200 is an example of a terminal. The voice input unit 219 is an example of a voice input unit. The wireless communication unit 223 is an example of a terminal communication unit. The CPU 211 is an example of a terminal control unit. The CPU 211 that executes the process of S306 is an example of a voice conversion unit. The CPU 211 that executes the process of S307 is an example of a text data transmission unit.

  Although the present invention has been described based on the embodiments, the present invention is not limited to the above-described embodiments, and various improvements and modifications can be easily made without departing from the spirit of the present invention. Can be inferred.

  For example, in the above-described embodiment, the MFP 10 that is a multi-function peripheral device is exemplified as the processing execution device. However, if the device is configured to execute processing using a button displayed on the screen, an appropriate device, for example, A device having a single function such as a scanner device or a printer device, a device for reproducing images and music, and the like can be adopted as the processing execution device of the present invention.

  In the above embodiment, as an example of a button operated by text data received from the terminal 200, a button including an image and a character string associated with the image is illustrated, but a button configured without including an image May be configured to be operated. For example, the present invention may be applied to a configuration in which one button (that is, one character string) is selected by moving the cursor among character strings as buttons arranged in a list.

  In the above embodiment, the tablet terminal is exemplified as the terminal 200 that performs remote operation of the MFP 10 by transmitting text data. However, the terminal 200 may be a personal computer, a smartphone, a mobile phone, a digital camera, a portable music playback device, or the like. There may be. In the above embodiment, the communication between the MFP 10 and the terminal 200 is exemplified by Wi-Fi communication in the infrastructure mode via the AP 50, but Wi-Fi communication in the ad hoc mode that is a direct connection, or Bluetooth. The present invention can also be applied to other communication methods such as (registered trademark).

  In the above embodiment, the text data converted from the voice input to the terminal 200 is exemplified as the text data received by the MFP 10 from the terminal 200. However, the text data is not limited to this. For example, the configuration may be such that text data corresponding to a character string input to the terminal 200 is transmitted to the MFP 10 using a device capable of inputting characters such as a keyboard and a pen tablet.

  In the above embodiment, the language corresponding to the conversion from the input speech has been described as English, but another language may be used. In the case of conversion to characters that do not correspond to uppercase / lowercase characters, such as conversion from input speech to kana characters, the processing of S610 to S612 and S616 to S618 can be omitted. In the case of conversion to characters that do not correspond to full-width / half-width, the processing of S613 to S615 can be omitted.

  In the above-described embodiment, the terminal 200 is configured to obtain one text character string by converting the input speech in S306. However, when there are a plurality of converted text character string candidates, the plurality of candidate character strings are converted. A configuration may be adopted in which all text data is transmitted to the MFP 10.

  In the collation process (S504), when the acquired text and the screen text are completely matched, the matching is determined. On the other hand, not only a perfect match, but also a configuration may be adopted in which a match is determined when the acquired text and the screen text match at a matching rate of a predetermined ratio or higher (for example, a matching rate of 80% or higher). Further, if the acquired text is a character string related to “Fax” that is the screen text, such as “Facsimile”, it may be determined that the two texts match.

  In the collation process (S504), in S601, all the character strings included in the screen being displayed on the LCD 16 are targeted for retrieval. However, only the character strings of the buttons may be extracted and retrieved. Alternatively, when the screen being displayed is a text input screen, all character strings included in the screen being displayed on the LCD 16 are to be searched, and the screen being displayed is a screen other than the text input screen. May be extracted by extracting only the character string of the button.

  In the above-described embodiment and modification, the CPU 11 has been described as a configuration for executing each process described in FIGS. 4 to 6. However, a plurality of CPUs cooperatively execute each process described in these figures. It is good also as composition to do. Further, an IC such as an ASIC may be configured to execute each process described in each of the above drawings independently or in cooperation with a plurality of ICs. Further, the CPU 11 and an IC such as an ASIC may cooperate to execute each process described in the above drawings. Moreover, you may abbreviate | omit or change a one part process in each process shown to FIGS. 4-6 in the range which does not deviate from the independent claim in a claim.

10: MFP. 12a: Control program

Claims (11)

A processing execution device;
A terminal,
A process execution system comprising:
The terminal
A voice input unit;
A terminal communication unit that communicates with the processing execution device;
A terminal control unit,
The terminal control unit
Voice conversion means for converting the voice into text data when voice is input via the voice input unit;
Text data transmission means for transmitting the text data converted by the voice conversion means to the processing execution device via the terminal communication unit;
With
The process execution device includes:
A display where the screen is displayed;
A storage unit;
A communication unit for communicating with the terminal;
A control unit;
With
The screen includes an operator associated with the process,
The storage unit stores, for each of the operators, text data associated with the operator,
The controller is
Text determination means for determining whether text data corresponding to text data received from a terminal is stored in the storage unit via the communication unit;
When the text determination unit determines that text data corresponding to the received text data is stored in the storage unit, a process execution is performed to execute a process according to an operator associated with the text data Means,
First screen determination means for determining whether the screen being displayed on the display unit is a screen for inputting text;
When the first screen determining unit determines that the screen being displayed is a screen for inputting text, the process executing unit based on the text data received from the terminal does not execute the process. Text receiving means for receiving the received text data as text input to the displayed screen;
A processing execution system comprising: The storage unit included in the processing execution device includes, for each operator, image data corresponding to a first image indicating the operator, text data associated with the operator, and text based on the text data. Associate and remember attributes related to visibility
The control unit provided in the processing execution device,
When an attribute for displaying text is associated with an operator included in a screen included in a display target, the first image and text based on text data corresponding to the operator are associated with the operator. When an attribute for hiding text is associated with an operator included in the screen included in the display target while being displayed on the display target screen, the corresponding operator is associated with the operator. The processing execution system according to claim 1, further comprising display means for displaying the first image on the display target screen without displaying text based on the text data to be displayed. The screen included in the processing execution device includes a background image and the operation element displayed on the background image,
The display means included in the process execution device, when an attribute for displaying text is associated with an operator included in a screen included in a display target, for the operator, the first image and the operation The text based on the text data corresponding to the child is displayed on the background image of the display target screen, while the attribute for hiding the text is associated with the operator included in the screen included in the display target. If it is, the first image is displayed on the background image of the display target screen without displaying the text based on the text data corresponding to the operator. Item 3. The processing execution system according to Item 2. The screen for inputting the text included in the processing execution device includes a screen in which the character type of the input text is not limited, and a screen in which the character type of the input text is limited,
The control unit provided in the processing execution device,
When the first screen determining means determines that the displayed screen is a screen for inputting text, the text data received from the terminal is composed of character types that can be input to the displayed screen. A character type judging means for judging whether or not
If it is determined by the character type determination means that the received text data is not composed of character types that can be input to the screen being displayed, the first error information is sent to the terminal that has output the text data. First error output means for outputting;
With
The text receiving means included in the processing execution device is determined by the character type determining means that the text data received from the terminal is determined to be composed of character types that can be input to the screen being displayed. process execution system according to any one of claims 1 to 3, as a condition, wherein the receiving the text data the received. The process execution means included in the process execution device is determined by the first screen determination means that the screen being displayed is a screen for inputting text, and received from the terminal by the text determination means If it is determined that text data corresponding to the text data is stored in the storage unit, a process corresponding to the operator associated with the received text data is executed,
The text receiving means included in the processing execution device is determined by the first screen determining means that the screen being displayed is a screen for inputting text, and the received text data by the text determining means. If the corresponding text data is not stored in the storage unit, the processing execution system according to any one of claims 1 to 4, characterized in that accepts text data the received. The control unit provided in the processing execution device,
Second screen determination means for determining whether the specific information received from the terminal together with the text data and specifying the screen displayed on the terminal specifies the screen being displayed on the display unit;
When the second screen determining unit determines that the received specific information does not specify the screen being displayed on the display unit, the processing by the process executing unit based on the text data received together with the specific information 2nd error which outputs the 2nd error information which shows that the screen currently displayed on the said display part differs from the screen currently displayed on the said terminal with respect to the terminal which output the said text data, without performing Output means;
The processing execution system according to any one of claims 1 to 5 , further comprising: The control unit provided in the processing execution device,
When the text determination unit determines that text data corresponding to the text data received from the terminal is not stored in the storage unit, an operation corresponding to the text data is performed on the terminal that has output the text data. process execution system according to any one of claims 1 to 6, characterized in that the third has an error output means for outputting a third error information indicating that there are no children. The text determination means included in the process execution device includes:
First determination means for determining whether text data corresponding to text data received from the terminal is stored in the storage unit;
When the first determination means determines that text data corresponding to the received text data is not stored in the storage unit, the received text data or the text data stored in the storage unit A second determination unit that changes a combination of uppercase and lowercase letters and / or full-width and half-width, and determines whether the changed text data corresponds to the other text data,
The text determination unit included in the processing execution device may be configured such that the first determination unit determines that text data corresponding to the received text data is stored in the storage unit, or the second determination. When it is determined by the means that the changed text data corresponds to the other text data, it is determined that the text data corresponding to the received text data is stored in the storage unit. A processing execution system according to any one of claims 1 to 7 . The control unit provided in the processing execution device,
During the animation display contents change over time is displayed on the display unit, one of claims 1 8, characterized in that it comprises a disabling means for disabling the text data received from the terminal The processing execution system according to the above. A display where the screen is displayed;
A storage unit;
A communication unit that communicates with the terminal;
A control unit;
With
The screen includes an operator associated with the process,
The storage unit stores, for each of the operators, text data associated with the operator,
The controller is
Text determination means for determining whether text data corresponding to text data received from the terminal is stored in the storage unit via the communication unit;
When the text determination unit determines that text data corresponding to the received text data is stored in the storage unit, a process execution is performed to execute a process according to an operator associated with the text data Means,
First screen determination means for determining whether the screen being displayed on the display unit is a screen for inputting text;
When the first screen determining unit determines that the screen being displayed is a screen for inputting text, the process executing unit based on the text data received from the terminal does not execute the process. Text receiving means for receiving the received text data as text input to the displayed screen;
A process execution device comprising: A display where the screen is displayed;
A storage unit;
A communication unit that communicates with the terminal;
A processing execution program that can be executed by the control unit.
The screen includes an operator associated with the process,
The storage unit stores, for each of the operators, text data associated with the operator,
Display means for displaying a screen on the display unit;
Text determination means for determining whether text data corresponding to text data received from the terminal is stored in the storage unit via the communication unit;
When the text determination unit determines that text data corresponding to the received text data is stored in the storage unit, a process execution is performed to execute a process according to an operator associated with the text data Means,
First screen determination means for determining whether the screen being displayed on the display unit is a screen for inputting text;
When the first screen determining unit determines that the screen being displayed is a screen for inputting text, the process executing unit based on the text data received from the terminal does not execute the process. As text accepting means for accepting received text data as text input to the screen being displayed,
A processing execution program for causing the control unit to function.