JP2019186766A - Image processing device, operation support method, and operation support program - Google Patents

Abstract

To support the operation of an image processing device by using information displayed by a smart phone.SOLUTION: An image processing device can communicate with a smart phone and includes: a display unit that displays an operation screen: an image acquisition unit 51 that acquires a terminal image displayed by the smart phone from the smart phone; and an operation support unit 55 that changes the contents of the operation screen on the basis of information included in the acquired terminal image.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image processing device, an operation support method, and an operation support program, and more particularly to an image processing device that can be wirelessly connected to an external device, an operation support method and an operation support program executed by the image processing device.

  Mobile devices such as smartphones and PDAs (Personal Digital Assistants) carried by users have a rich user interface. For example, in addition to inputting character information using a keyboard, it is possible to input still images, moving images, and sounds. In addition, the portable device has a rich communication function and stores personal data such as an address book.

  On the other hand, image processing apparatuses such as an MFP (Multi Function Peripheral) have become multifunctional, and operations have become complicated. For this reason, in order to support the user who operates the MFP, it is conceivable to install an application program for supporting the operation of the MFP in the portable device.

  However, since the MFP and the portable device are separate devices, even if the information for supporting the operation of the MFP is displayed on the display and output by the portable device, the user displays the information on the MFP based on the display. There is a problem that the user cannot immediately understand the operation on the operation screen.

As a technology for supporting a user who carries a portable device, Japanese Patent Application Laid-Open No. 2006-134042 discloses a technology for transmitting data obtained by reading an original with an MFP to a destination defined in an address book stored in the portable device. Is described. However, the technique described in Japanese Patent Application Laid-Open No. 2006-134042 requires that a procedure and information for communicating between the portable device and the MFP be determined in advance. For this reason, there is a problem that it is necessary to use a portable device in which a dedicated application program in which a procedure for cooperation with the MFP is defined is installed, and cooperation can be performed only with a predetermined function.
Japanese Patent Laying-Open No. 2006-134042

  The present invention has been made to solve the above-described problems, and one object of the present invention is to provide an image processing apparatus capable of supporting an operation using information displayed by a portable device. It is.

  Another object of the present invention is to provide an operation support method capable of supporting an operation of an image processing apparatus using information displayed by a portable device.

  Another object of the present invention is to provide an operation support program capable of supporting an operation of an image processing apparatus using information displayed by a portable device.

  In order to achieve the above-described object, according to one aspect of the present invention, an image processing apparatus is an image processing apparatus capable of communicating with a portable device, and includes a display unit that displays an operation screen, and the portable device to the portable device. Image acquisition means for acquiring the terminal image being displayed, and operation support means for changing the content of the operation screen based on information included in the acquired terminal image.

  According to this aspect, the content of the operation screen can be matched with the information included in the terminal image acquired from the portable device. As a result, it is possible to provide an image processing apparatus capable of supporting an operation using information displayed by the portable device.

  Preferably, the operation screen includes a display component for accepting an operation input by the user, and the operation support means displays a target of an operation scheduled to be input by the user based on information included in the terminal image. Prediction means for predicting a part as a target part; and highlighting means for highlighting and displaying the predicted target part.

  According to this aspect, based on the result of analyzing the terminal image, the display component that is the target of the operation that is scheduled to be input by the user is predicted as the target component, and the display component predicted by the target component is highlighted. To do. For this reason, the operation guided by the portable device can be notified.

  Preferably, the prediction means predicts the display component based on character information included in the terminal image.

  If this aspect is followed, since the display component used as the object of the operation planned to be input by the user is predicted based on the character information included in the terminal image, the accuracy of predicting the display component can be improved.

  Preferably, the predicting unit compares the character string included in the character information with a reserved word associated with the display component in advance.

  If this situation is followed, since the character string contained in character information is compared with the reserved word previously linked | related with the display component, the precision which estimates a display component can be improved.

  Preferably, the predicting unit predicts the display component based on a character string that frequently appears from a plurality of character strings included in the character information.

  According to this aspect, since the display component is predicted based on a character string that frequently appears from among a plurality of character strings included in the character information, the accuracy of predicting the display component can be improved.

  Preferably, the prediction unit predicts a display component similar to an image included in the terminal image as a target component.

  According to this aspect, since a display component similar to the image included in the terminal image is predicted as the target component, the accuracy of predicting the display component can be increased.

  Preferably, when the terminal image includes a plurality of images that are respectively similar to the plurality of display components, the prediction unit predicts a display component that is similar to a high-frequency image among the plurality of images as a target component.

  According to this aspect, when the terminal image includes a plurality of similar images that are respectively similar to the plurality of display components, a similar image having a high frequency is extracted, so that the accuracy of predicting the display components can be improved.

  Preferably, the highlighting means displays the terminal image in the vicinity of the display component for receiving the predicted operation.

  According to this aspect, since the terminal image is displayed in the vicinity of the predicted display component, the position of the predicted display component on the operation screen can be notified.

  Preferably, when the operation screen and the terminal image are displayed on the display unit before the target component is predicted, the highlighting unit moves the position where the terminal image is displayed.

  According to this aspect, when the operation screen and the terminal image are displayed side by side before the target component is predicted, the position where the terminal image is displayed is moved. The predicted display component can be notified.

  Preferably, the highlighting means moves the display component predicted as the target component to a position prioritized over other display components.

  According to this aspect, since the predicted display component is moved to a position that is prioritized over other display components, the display component can be notified at the priority position on the operation screen.

  Preferably, the display unit displays the terminal image in addition to the operation screen, and the highlighting unit includes the display component predicted as the target component and the original information included in the terminal image and from which the target component is predicted. Highlight the area where is displayed.

  According to this aspect, since the display component predicted as the target component and the area where the original information included in the terminal image and from which the display component is predicted are displayed are highlighted, the display component and the original information are displayed. Can be notified of the relationship.

  Preferably, the highlighting means displays on the display means an instruction image that associates the display part predicted as the target part with the area where the original information is displayed.

  According to this aspect, an instruction image is displayed that associates the display component predicted to the target component with the area where the original information is displayed. Therefore, the display component is notified to the user, and the original information related to the display component is displayed. You can be notified.

  Preferably, the highlighting means does not display the terminal image when the display component is not predicted as the target component, or reduces the size of the terminal image when the terminal image is displayed.

  According to this aspect, when the display component is not predicted, the terminal image is not displayed, and when the terminal image is displayed, the size of the terminal image is reduced, so that the terminal image is related to the display component. It can be notified that no information exists.

  According to another aspect of the present invention, the operation support method includes an image acquisition step of acquiring a terminal image displayed by the portable device from the portable device, and display means based on information included in the acquired terminal image. The image processing apparatus executes an operation support step for changing the content of the operation screen displayed on the screen.

  If this aspect is followed, the operation assistance method which can assist operation of an image processing apparatus using the information which a portable apparatus displays can be provided.

  According to still another aspect of the present invention, the operation support program is an operation support program executed by a computer that controls the image processing apparatus, and an image for acquiring a terminal image displayed by the mobile apparatus from the mobile apparatus. Based on the information included in the acquired terminal image, the acquisition step and the operation support step of changing the content of the operation screen displayed on the display means are executed by the computer.

  If this aspect is followed, the operation assistance method which can assist operation of an image processing apparatus using the information which a portable apparatus displays can be provided.

It is a figure showing an example of the whole outline of an information system in an embodiment of the invention. 1 is a perspective view showing an external appearance of an MFP in the present embodiment. 2 is a block diagram illustrating an example of a hardware configuration of an MFP according to the present embodiment. FIG. It is a figure which shows an example of a detailed structure of an NFC communication part. It is a top view of an operation panel. It is a block diagram which shows the outline | summary of the hardware constitutions of the smart phone in this Embodiment. 3 is a block diagram showing an example of functions of a CPU provided in the MFP according to the present embodiment. FIG. It is a flowchart which shows an example of the flow of an operation assistance process. It is a flowchart which shows an example of the flow of a prediction process. It is a flowchart which shows an example of the flow of an operation screen change process. It is a flowchart which shows an example of the flow of an operation reception process. FIG. 3 is a first diagram illustrating an example of an image displayed on an MFP. FIG. 10 is a second diagram illustrating an example of an image displayed on the MFP. FIG. 10 is a third diagram illustrating an example of an image displayed on the MFP. FIG. 10 is a fourth diagram illustrating an example of an image displayed on the MFP. FIG. 10 is a fifth diagram illustrating an example of an image displayed on the MFP. FIG. 10 is a sixth diagram illustrating an example of an image displayed on the MFP.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  FIG. 1 is a diagram showing an example of an overall outline of an information system according to an embodiment of the present invention. Referring to FIG. 1, information system 1 includes an MFP (Multi Function Peripheral) 100 and a smartphone 200.

  The smartphone 200 is an example of a portable device, and is used by being carried by a user. Since the hardware configuration and functions of smartphone 200 are well known, description thereof will not be repeated here.

  MFP 100 is an example of an image processing apparatus, and has a document reading function for reading a document, an image forming function for forming an image on a recording medium such as paper based on image data, and a facsimile transmission / reception function for transmitting / receiving facsimile data. I have.

  Each of the wireless station 5 and the MFP 100 is connected to the network 3. The network 3 is a local area network (LAN), and the connection form may be wired or wireless. In addition, the network 3 is not limited to a LAN, and may be a network using a public switched telephone network (Public Switched Telephone Networks). Further, the network 3 may be a wide area network (WAN) such as the Internet.

  The smartphone 200 has a wireless LAN function and can communicate with the wireless station 5. The wireless station 5 is a relay device of the network 3, communicates with the smartphone 200 having a communication function using a wireless LAN, and connects the smartphone 200 to the network 3. For this reason, the smartphone 200 can communicate with the MFP 100 via the wireless station 5. The network 3 may be further connected to the Internet. In this case, each of the smartphone 200 and the MFP 100 can communicate with a computer connected to the Internet via the network 3.

  Furthermore, each of MFP 100 and smartphone 200 has a function of performing wireless communication at a short distance, and can communicate with each other on a one-to-one basis when the counterpart device is within a predetermined range.

  FIG. 2 is a perspective view showing an appearance of the MFP according to the present embodiment. FIG. 3 is a block diagram illustrating an example of the hardware configuration of the MFP according to the present embodiment. 2 and 3, MFP 100 includes a main circuit 110, a document reading unit 130 for reading a document, an automatic document transport device 120 for transporting a document to document reading unit 130, and a document reading unit. An image is formed by an image forming unit 140 for forming an image on a sheet or the like based on image data output by the manuscript 130 reading and outputting a document, a paper feeding unit 150 for supplying the image to the image forming unit 140, and an image. A post-processing unit 155 that processes paper and an operation panel 160 as a user interface are included.

  The post-processing unit 155 executes a sorting process for sorting and discharging one or more sheets on which images are formed by the image forming unit 140, a punching process for punch hole processing, and a staple process for driving staple needles.

  The main circuit 110 includes a CPU 111, a communication interface (I / F) unit 112, a ROM 113, a RAM 114, a hard disk drive (HDD) 115, a facsimile unit 116, an external storage device 117, and an NFC communication unit 119 connected to the CPU 111. Including. CPU 111 is connected to automatic document feeder 120, document reading unit 130, image forming unit 140, paper feeding unit 150, post-processing unit 155, and operation panel 160, and controls the entire MFP 100.

  The ROM 113 stores a program executed by the CPU 111 or data necessary for executing the program. The RAM 114 is used as a work area when the CPU 111 executes a program. The RAM 114 temporarily stores read data (image data) continuously sent from the document reading unit 130.

  Operation panel 160 is provided on the upper surface of MFP 100 and includes a display unit 161 and an operation unit 163. The display unit 161 is a display device such as a liquid crystal display (LCD) or an organic ELD (Electro-Luminescence Display), and displays an instruction menu for the user, information about acquired image data, and the like. The operation unit 163 includes a hard key unit 167 including a plurality of keys, and accepts input of various instructions, data such as characters and numbers by user operations corresponding to the keys. The operation unit 163 further includes a touch panel 165 provided on the display unit 161.

  The HDD 115 is a mass storage device. Communication I / F unit 112 is an interface for connecting MFP 100 to network 3. The CPU 111 communicates with the smartphone 200 via the communication I / F unit 112 to transmit / receive data. The communication I / F unit 112 can communicate with a computer connected to the Internet via the network 3.

  The facsimile unit 116 is connected to the public switched telephone network (PSTN) and transmits facsimile data to the PSTN or receives facsimile data from the PSTN. The facsimile unit 116 stores the received facsimile data in the HDD 115 or outputs it to the image forming unit 140. The image forming unit 140 prints the facsimile data received by the facsimile unit 116 on a sheet. Further, the facsimile unit 116 converts the data stored in the HDD 115 into facsimile data, and transmits the facsimile data to a facsimile machine connected to the PSTN.

  The external storage device 117 is loaded with a CD-ROM (Compact Disk ROM) 118. The CPU 111 can access the CD-ROM 118 via the external storage device 117. The CPU 111 loads the program recorded on the CD-ROM 118 attached to the external storage device 117 to the RAM 114 and executes it. The medium for storing the program executed by the CPU 111 is not limited to the CD-ROM 118, but an optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC card, optical card, It may be a semiconductor memory such as a mask ROM or EPROM (Erasable Programmable ROM).

  Further, the program executed by the CPU 111 is not limited to the program recorded on the CD-ROM 118, and the program stored in the HDD 115 may be loaded into the RAM 114 and executed. In this case, another computer connected to the network 3 may rewrite the program stored in the HDD 115 of the MFP 100 or may add a new program and write it. Further, MFP 100 may download a program from another computer connected to network 3 and store the program in HDD 115. The program here includes not only a program directly executable by the CPU 111 but also a source program, a compressed program, an encrypted program, and the like.

  FIG. 4 is a diagram illustrating an example of a detailed configuration of the NFC communication unit. Referring to FIG. 4, NFC communication unit 119 includes an NFC communication control unit 173, an NFC antenna 171, and a detection circuit 175. The NFC communication control unit 173 is connected to the NFC antenna 171. The NFC communication control unit 173 is controlled by the CPU 111, and communicates with an external device, for example, the smartphone 200 wirelessly by an NFC (Near Field Communication) near field communication method. The distance that the NFC communication unit 119 can communicate is several centimeters. The NFC antenna 171 is disposed on a surface above or below the display surface of the display unit 161. Specifically, the NFC antenna 171 may be formed on a substrate disposed below the display unit 161 or may be disposed above the display surface of the display unit 161. When the NFC antenna 171 is disposed above the display surface of the display unit 161, the NFC antenna 171 is formed of a transparent material. Further, when the display unit 161 is disposed above the display surface, the display unit 161 may be disposed between the display unit 161 and the touch panel 165 or may be disposed on the touch panel 165.

  When the NFC antenna 171 receives a radio wave having a predetermined frequency for the NFC communication unit 119 to communicate, the detection circuit 175 outputs a signal indicating that communication with an external device is possible to the CPU 111. .

  FIG. 5 is a plan view of the operation panel. Referring to FIG. 5, operation panel 160 includes a display unit 161, a touch panel 165, and a hard key unit 167. The display surface of the display unit 161 and the detection surface of the touch panel 165 are arranged so as to overlap each other. The antenna region 171A is a region overlapping with the NFC antenna 171 in the display surface of the display unit 161.

  Hard key portion 167 includes a plurality of hard keys. The hard key is a contact switch and is connected to the CPU 111. The hard key is OFF when not pressed by the user, and is turned ON when pressed by the user.

  FIG. 6 is a block diagram illustrating an outline of the hardware configuration of the smartphone according to the present embodiment. Referring to FIG. 6, smartphone 200 in the present embodiment is connected to CPU 201 for controlling entire smartphone 200, camera 202, flash memory 203 for storing data in a nonvolatile manner, and call unit 205. A wireless communication unit 204, a display unit 206 for displaying information, an operation unit 207 for accepting user operations, a wireless LAN I / F 208, an NFC communication unit 209, and an external storage device 211.

  The display unit 206 is a display device such as a liquid crystal display (LCD) or an organic ELD, and displays an instruction menu for a user, information about acquired image data, and the like. The operation unit 207 includes a main key 207A and a touch panel 207B. When the user instructs the display surface of the display unit 206, the operation unit 207 outputs the position of the display surface detected by the touch panel 207B to the CPU 201. Based on the position detected by the touch panel 207B, the CPU 201 detects a position designated by the user in the screen displayed on the display unit 206. Based on the screen displayed on the display unit 206 and the position detected by the touch panel 207 </ b> B, the CPU 201 accepts input of various instructions, data such as characters and numbers by user operations. For example, when a screen including a numeric keypad image is displayed on the display unit 206, a number corresponding to the key displayed at the position detected by the touch panel 207B is received.

  The camera 202 includes a lens and a photoelectric conversion element. The light focused by the lens is imaged on the photoelectric conversion element. The photoelectric conversion element photoelectrically converts the received light and outputs image data to the CPU 201. A photoelectric conversion element is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like.

  The wireless communication unit 204 performs wireless communication with a mobile phone base station connected to a telephone communication network. The wireless communication unit 204 connects the smartphone 200 to the telephone communication network and enables a call using the call unit 205. The radio communication unit 204 decodes a voice signal obtained by demodulating the radio signal received from the mobile phone base station and outputs the decoded signal to the call unit 205. Further, the wireless communication unit 204 encodes the voice input from the call unit 205 and transmits it to the mobile phone base station. The call unit 205 includes a microphone and a speaker, and outputs sound input from the wireless communication unit 204 from the speaker, and outputs sound input from the microphone to the wireless communication unit 204. Further, the wireless communication unit 204 is controlled by the CPU 201, connects the smartphone 200 to an email server, and transmits and receives emails.

  The wireless LAN I / F 208 is an interface for communicating with the wireless station 5 and connecting the smartphone 200 to the network 3. By registering the IP (Internet Protocol) address of the MFP 100 in the smartphone 200, the smartphone 200 can communicate with the MFP 100 and transmit and receive data.

  NFC communication unit 209 communicates with MFP 100 wirelessly using an NFC short-range wireless communication method. For example, the NFC communication unit 209 communicates with the MFP 100 when the distance to the MFP 100 is equal to or less than the communicable distance. The distance that the NFC communication unit 209 can communicate is several centimeters.

  The flash memory 203 stores a program executed by the CPU 201 or data necessary for executing the program. The CPU 201 loads the program recorded in the flash memory 203 to the RAM included in the CPU 201 and executes it.

  The external storage device 211 is detachably attached to the smartphone 200, and a CD-ROM 211A storing an application program can be attached thereto. The CPU 201 can access the CD-ROM 211A via the external storage device 211. The CPU 201 can load an application program recorded on the CD-ROM 211A mounted on the external storage device 211 into a RAM included in the CPU 201 and execute it.

  Although the program recorded in the flash memory 203 or the CD-ROM 211A has been described as a program executed by the CPU 201, a program in which another computer connected to the network 3 rewrites the program stored in the flash memory 203, Alternatively, it may be a new program written additionally. Furthermore, the program which the smart phone 200 downloaded from the other computer connected to the network 3 may be sufficient. The program here includes not only a program directly executable by the CPU 201 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 201 is not limited to the CD-ROM 211A, and may be a semiconductor memory such as an optical disk (MO / MD / DVD), IC card, optical card, mask ROM, EPROM, or EEPROM. Good.

  Smartphone 200 according to the present embodiment is installed with a support information providing program that provides support information for supporting the operation of MFP 100 as an application program. For example, when the CPU 111 executes the support information providing program, the task executing the support information providing program displays support information associated with the character information input by the user on the display unit 206. The character information is preferably a keyword for searching for support information, and is, for example, the name of a process that the user desires to operate. The support information is information that supports the operation of the MFP 100. For example, the support information is information describing an operation on the operation screen displayed on the MFP 100. Further, in order to support an operation for setting a set value, information indicating an area for inputting a set value in the operation screen may be included, or included in the operation screen to support an operation for switching the operation screen. Information indicating a button may be used.

  In order to receive input of character information, the smartphone 200 receives character information input to the operation unit 207 by the user. When a voice recognition program for recognizing voice is installed in the smartphone 200, the smartphone 200 receives a voice uttered by the user with a microphone included in the calling unit 205, and executes a task for executing the voice recognition program. Character information may be received by converting voice into character information.

  The operation support program may have a learning function. For example, history information in which character information input by the user of the smartphone 200 in the past and support information selected by the user from among a plurality of support information corresponding to the character information is stored, and the history information is stored. Based on the above, appropriate support information corresponding to the character information input by the user is presented.

  Moreover, the application program installed in the smartphone 200 is not limited to the support information providing program, and may be a program installed in the smartphone 200 in order to facilitate the operation of the smartphone 200. For example, when the user inputs his / her name by voice, the program may display the address information of the user specified by the name from the address book stored in the smartphone 200.

  FIG. 7 is a block diagram illustrating an example of functions of the CPU provided in the MFP according to the present embodiment. The functions illustrated in FIG. 7 are functions formed in the CPU 111 when the CPU 111 provided in the MFP 100 executes a setting value setting program stored in the ROM 113, HDD 115, or CD-ROM 118. Each function of the CPU 111 shown in FIG. 7 executes a predetermined process for the input and outputs a process result. Therefore, each function of the CPU 111 can be realized by an electronic circuit using a relay or the like. Referring to FIG. 7, CPU 111 includes image acquisition unit 51, analysis unit 53, operation support unit 55, terminal image display unit 57, operation reception unit 59, operation screen display unit 61, and set value determination. Unit 63 and process execution unit 65.

  The image acquisition unit 51 communicates with the smartphone 200 and acquires an image displayed on the display unit 206 by the smartphone 200. For example, when the image acquisition unit 51 detects that the NFC communication unit 119 can communicate with the smartphone 200, the image acquisition unit 51 controls the communication I / F unit 112 to establish a communication path with the smartphone 200. The NFC communication unit 119 has a predetermined frequency of radio waves for communication with the smartphone 200. For this reason, the NFC communication part 119 becomes communicable with the smart phone 200, when receiving the electromagnetic wave of a predetermined frequency.

  At this stage, when a general-purpose application program that defines a process for transmitting an image displayed on the display unit 206 is executed in the smartphone 200, the communication I / F unit 112 receives an image transmitted by the smartphone 200. To do. The image acquisition unit 51 acquires an image received by the communication I / F unit 112 from the smartphone 200 as a terminal image. The image acquisition unit 51 outputs the terminal image acquired from the smartphone 200 to the analysis unit 53 and the terminal image display unit 57. Therefore, the terminal image can be acquired from the smartphone 200 without communicating with the MFP 100 on the smartphone 200 and installing a dedicated application program that matches the specifications of the MFP 100.

  The image acquisition unit 51 acquires the stream data that the communication I / F unit 112 receives from the smartphone 200 when the smartphone 200 transmits stream data. In this case, the image acquisition unit 51 extracts one frame of a still image from the stream data, and acquires the extracted still image as a terminal image.

  The operation screen display unit 61 controls the display unit 161 to display the operation screen stored in the HDD 115 in a predetermined first area on the display surface of the display unit 161. As a result, the user can operate the operation unit 163 to input an operation to the MFP 100. The operation screen includes a screen for setting a setting value for the user to cause MFP 100 to execute a process. The set value is a value used by the process execution unit 65 to execute the process. The operation screen display unit 61 outputs screen identification information for identifying the displayed operation screen to the operation support unit 55 and the set value determination unit 63 in response to displaying the operation screen on the display unit 161.

  The operation screen includes one or more display components. The display component includes a notification component that displays information to be notified to the user, an input component that receives a value input by the user, and an instruction component that receives an operation that instructs execution of processing. The display component is not limited to these.

  The notification component includes, for example, an explanatory component that represents a character string that describes the operation screen, a message component that represents a message that prompts the next operation, and the like. The input component includes a direct reception component that receives a setting value input by the user, a selection reception component that receives an operation for selecting one or more of a plurality of setting values, and a command associated with a command for executing a predetermined process Including parts. The direct reception component is, for example, a display component for receiving setting values directly input by the user, such as the number of copies. The selection receiving component is a display component for receiving an operation of selecting one or more from a plurality of predetermined setting values such as a paper size. The command component is a display component associated with a command or the like that transitions to a specific operation screen.

  The display component may be included in the operation screen as an image such as an icon. For example, it is an image that accepts an operation for scrolling the screen.

  The terminal image display unit 57 displays the terminal image on the display unit 161 in response to the input of the terminal image. The terminal image display unit 57 displays a terminal image in a predetermined second area on the display surface of the display unit 161. The second area does not overlap with the first area.

  The operation reception unit 59 controls the operation unit 163 to receive an operation input by the user to the operation unit 163. The operation accepting unit 59 outputs the accepted operation to the operation screen display unit 61 and the set value determination unit 63.

  When the user inputs a setting value to the operation unit 163 according to the operation screen displayed on the display unit 161, the operation reception unit 59 receives a setting operation for setting the setting value. When the setting operation is received by the operation receiving unit 59, the setting value determining unit 63 determines a setting value specified by the received setting operation. The set value determining unit 63 outputs the determined set value to the process executing unit 65. Specifically, when the user designates an area for inputting a setting value in the operation screen, the setting value determining unit 63 detects a position in the area in which the touch panel 165 inputs the setting value in the operation screen. An instruction operation is accepted. When accepting the area instruction operation, the setting value determination unit 63 sets the type of the setting value corresponding to the area specified by the area instruction operation as a processing target. Then, when the user inputs a setting value to the operation unit 163, the setting value determination unit 63 determines the setting value received by the operation unit 163 as the setting value of the type set as the processing target. For example, an arrangement of numbers input by the user by operating the hard key unit 167 is determined as the set value. In addition, a keyboard is displayed on the display unit 161, characters assigned to keys corresponding to the positions detected by the touch panel 165 are received, and an array of a plurality of received characters is determined as a set value.

  When the operation accepting unit 59 accepts an execution instruction operation, the process executing unit 65 executes the process according to the set value determined by the set value determining unit 63. The operation reception unit 59 receives an execution instruction operation when inputting an operation to instruct the operation unit 163 to execute the process. The processes that can be executed by the process execution unit 65 include, for example, a document reading process, an image forming process, a data transmission / reception process, a facsimile transmission / reception process, and a data management process. The processing that can be executed by the processing execution unit 65 includes processing combining at least two of document reading processing, image formation processing, data transmission / reception processing, facsimile transmission / reception processing, and data management processing. The processes that can be executed by the process execution unit 65 are not limited to these, and may be smaller or may include more types of processes.

  The analysis unit 53 receives the terminal image from the image acquisition unit 51 and analyzes the terminal image. The analysis unit 53 analyzes the terminal image in response to the terminal image input from the image acquisition unit 51. In other words, the analysis unit 53 analyzes the terminal image every time a new terminal image is input from the image acquisition unit 51. The analysis unit 53 may analyze the terminal image in response to the operation reception unit 59 receiving a start instruction operation. The operation receiving unit 59 receives a start instruction operation when the user inputs an operation for instructing the operation unit 163 to start analysis. For example, when the terminal image is displayed on the display unit 161 by the terminal image display unit 57, the operation reception unit 59 displays a start instruction button to which a start instruction is assigned on the display unit 161. When the position in the start instruction button is instructed by the user and the touch panel 165 detects the position in the start instruction button, the operation reception unit 59 receives a start instruction operation.

  The analysis unit 53 includes a character extraction unit 67 and an image extraction unit 69. The character extraction unit 67 recognizes characters in the terminal image and extracts characters. The character extraction unit 67 extracts characters from the entire terminal image and outputs character information including the extracted characters to the operation support unit 55. The image extraction unit 69 extracts an image other than characters from the terminal image. The image extraction unit 69 outputs the extracted image to the operation support unit 55.

  The operation support unit 55 includes a prediction unit 71 and a highlight display unit 75. The prediction unit 71 predicts a display component that is a target of a scheduled operation input by the user based on information included in the terminal image. The prediction unit 71 receives character information, an image, or a set of character information and an image from the analysis unit 53 as information included in the terminal image.

  The prediction unit 71 includes a comparison unit 73. The comparison unit 73 compares the character string included in the character information input from the character extraction unit 67 with a reserved word associated in advance with the display component. The reserved word is a part name given to a display part included in the operation screen and a word registered in advance for the display part. For example, for an input part for setting the number of copies, “part number”, “number of copies”, “number of copies”, etc., are used as words indicating the part name “number of copies” and the number of copy processing operations. It is registered in advance as a reserved word. In addition, for the selected reception component for setting the size of the document, “A4”, “A3”, “A4”, “A3”, and the “document size” that is the component name and words indicating one or more types of sheets stored in the MFP 100 Words indicating the paper size such as “B4” and words indicating the paper direction such as “vertical” and “horizontal” are registered in advance as reserved words. For the command component associated with the command component that transitions to the copy setting screen, which is the operation screen for setting the setting value for specifying the copy process, the component name "Copy setting screen", "Copy", Words such as “copy” are registered in advance as reserved words. The comparison unit 73 also compares the image input from the image extraction unit 69 with the image of the display component. Further, the reserved word may be added from an operation history by the user.

  As a result of the comparison by the comparison unit 73, the prediction unit 71 determines that the target character string included in the character information input from the character extraction unit 67 matches or is similar to the reserved word related to the display component. A display component related to a reserved word similar to a character string is predicted as a target component that is a target of a scheduled operation input by the user. In addition, as a result of comparison by the comparison unit 73, the prediction unit 71 determines that an image input from the image extraction unit 69 if the target image matches or resembles the image of the display component among the images input from the image extraction unit 69. A display component that matches or is similar to is predicted as a target component that is a target of an operation scheduled to be input by the user. The prediction unit 71 outputs the display component predicted as the target component to the highlight display unit 75.

  When there are a plurality of predicted display components, the prediction unit 71 inputs a display component corresponding to a character string or an image that appears most frequently in the terminal image among character strings or images extracted from the terminal image by the user. Predict as the target of the scheduled operation. This is because the higher the frequency of appearance in the terminal image, the higher the possibility of the display component being noted.

  The highlighting display unit 75 highlights and displays the display component specified by the component identification information input from the prediction unit 71 on the display unit 161. The emphasis display unit 75 includes a neighborhood arrangement unit 81, an arrangement change unit 83, a related display unit 85, and a reduction unit 87.

  The neighborhood arrangement unit 81 displays a terminal image in the vicinity of the display component specified by the component identification information in response to the input of the component identification information from the prediction unit 71. The neighborhood arrangement unit 81 displays the terminal image superimposed on the operation screen. For this reason, it is possible to notify the user of the predicted position of the display component on the operation screen. At this time, the terminal image may remain displayed in the second area, or the terminal image may not be displayed in the second area. Further, an image obtained by reducing the size of the terminal image may be displayed in the vicinity of the display component.

  When the terminal image is displayed in a predetermined second area on the display surface of the display unit 161, the vicinity arranging unit 81 displays the terminal image of the display component specified by the component identification information from the second area. You may make it move to the position of the vicinity. Moreover, you may make it move the image which copied the terminal image. The neighborhood arranging unit 81 may display a state in the middle of moving the terminal image from the second area to the vicinity of the display component. Thereby, the position of the display component can be notified to the user in an easy-to-understand manner.

  In response to the input of the component identification information from the prediction unit 71, the arrangement changing unit 83 sets the position where the display component specified by the component identification information is arranged as a position having a higher priority than other display components. The position where one or more display components in the operation screen are arranged is changed. Here, the priority order is the highest in the upper left of the operation screen, and the priority order decreases as the distance increases to the right or lower side. The arrangement changing unit 83 may display a state in which the display component moves from the position before the arrangement is changed to the position after the change. Thereby, the position of the display component can be notified to the user in an easy-to-understand manner.

  In response to the input of the component identification information from the prediction unit 71, the related display unit 85 emphasizes the set of the display component specified by the component identification information and the target character string or target image included in the terminal image. To display. Specifically, the related display unit 85 displays an instruction image that associates the display component with the target character string or target image on the display unit 161. For example, the instruction image is a line connecting a display component and a target character string or target image. By connecting the display component and the target character string or target image with a line, it is possible to notify the user that the display component is related to the target character string or target image and the display component.

  In addition, the instruction image may be a display component and a target character string, or two identical images arranged in the vicinity of the display component and the target image. By arranging the same two images, it is possible to notify the user that the display component arranged in the vicinity of each of the images is related to the target character string or the target image, and the display component.

  Any one of the neighborhood arrangement unit 81, the arrangement change unit 83, and the related display unit 85 functions. In this case, the change mode may be set in advance. For example, the change mode is set to any one of a superposition mode in which the neighboring arrangement unit 81 functions, an arrangement change mode in which the arrangement change unit 83 functions, and an instruction mode in which the related display unit 85 functions. The change mode may be selected by the user or may be preset in MFP 100. Furthermore, the vicinity arrangement unit 81, the arrangement change unit 83, and the related display unit 85 may be combined to function, or the vicinity arrangement unit 81, the arrangement change unit 83, and the related display unit 85 all function simultaneously. You may make it do.

  When the component identification information is not input from the prediction unit 71, the reduction unit 87 reduces the size of the terminal image and displays it in the second area. Accordingly, it is possible to notify the user that there is no information related to the display component in the terminal image. The reduction unit 87 may not display the terminal image on the display unit 161 when the component identification information is not input from the prediction unit 71.

  FIG. 8 is a flowchart illustrating an example of the flow of the operation support process. The operation support process is a process executed by CPU 111 when CPU 111 provided in MFP 100 executes an operation support program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 8, CPU 111 displays an operation screen on display unit 161 (step S01). One operation screen is displayed on the display unit 161 from the plurality of operation screens stored in the HDD 115. Of the plurality of operation screens, a predetermined operation screen may be displayed, or an operation screen determined by an operation input by the user to the operation unit 163 may be displayed.

  In the next step S02, it is determined whether or not the NFC communication unit 119 has detected another device. If the NFC communication unit 119 receives a radio wave having a predetermined frequency, it is determined that another device has been detected. For example, when the smartphone 200 carried by the user is brought close to the antenna of the NFC communication unit 119, the NFC communication unit 119 detects a radio wave having a frequency emitted from the smartphone 200, so the smartphone 200 is detected. The process waits until the NFC communication unit 119 detects another device (NO in step S02). If the NFC communication unit 119 detects another device (YES in step S02), the process proceeds to step S03. Hereinafter, a case where the NFC communication unit 119 detects the smartphone 200 will be described as an example.

  In step S03, a communication path is established with the device detected in step S02, here, smartphone 200, and the process proceeds to step S04. For example, the communication I / F unit 112 is controlled to establish a communication path with the smartphone 200. And it is judged whether the terminal image was received from the smart phone 200 (step S04). In the smart phone 200, when the application program which transmits the image displayed on the display part 206 is performed, the smart phone 200 transmits the image displayed on the display part 206 via the communication path established in step S03. Therefore, the communication I / F unit 112 receives an image transmitted from the smartphone 200 as a terminal image. If communication I / F unit 112 has received the terminal image, the process proceeds to step S05; otherwise, the process proceeds to step S08. In addition, when a communication path is not established with the smart phone 200 in step S03, since it cannot communicate with the smart phone 200, a process is advanced to step S08.

  In step S05, the terminal image is displayed in the second area of display unit 161, and the process proceeds to step S06. In step S06, a prediction process is executed, and the process proceeds to step S07. In step S07, an operation screen change process is executed, and the process proceeds to step S08. Details of the prediction process and the operation screen change process will be described later.

  In step S08, it is determined whether an operation has been accepted. If operation unit 163 accepts an operation input by the user, the process proceeds to step S09; otherwise, the process returns to step S04. In step S09, an operation acceptance process is executed, and the process proceeds to step S10. Details of the operation reception process will be described later. In step S10, it is determined whether or not the process is finished. If the process ends, the process ends. If not, the process returns to step S04.

  FIG. 9 is a flowchart illustrating an example of the flow of the prediction process. The prediction process is a process executed in step S06 in FIG. Referring to FIG. 9, CPU 111 recognizes characters in the terminal image (step S31), and advances the process to step S32. In step S32, a word to be processed is selected from character information extracted from the terminal image by character recognition, and the process proceeds to step S33.

  In step S33, it is determined whether there is a reserved word that is the same as or similar to the word to be processed. If such a reserved word exists, the process proceeds to step S34; otherwise, the process proceeds to step S35. In step S34, a word to be processed is determined as a candidate part, and a display component associated with a reserved word that is the same as or similar to the word to be processed is determined as a target component. The target part is a display part that is expected to be operated by the user. In the next step S35, it is determined whether or not the word not selected as the processing target in step S32 is included in the character information extracted from the target image. If there is an unselected word, the process returns to step S32. If not, the process proceeds to step S36. In step S36, it is determined whether there are a plurality of target parts. If there are a plurality of target parts, the process proceeds to step S37; otherwise, the process proceeds to step S38. In step S37, only the target part corresponding to the candidate part having the highest frequency of appearance in the character string extracted from the terminal image among the display parts determined as the target part in step S34 is operated by the user. A display component that is expected to be selected is selected, and the process proceeds to step S38. A display component corresponding to a candidate portion whose frequency of appearance in the character string extracted from the terminal image is not the maximum is excluded from the target components.

  In step S38, an image included in the terminal image is selected as a processing target. In the next step S39, it is determined whether or not the image selected as the processing target is similar to the image of the display component included in the operation screen. If the image selected as the processing target is the same as or similar to the image of the display component included in the operation screen, the process proceeds to step S40; otherwise, the process proceeds to step S41. In step S40, the processing target image is determined as a candidate portion, and a display component similar to the processing target image is determined as the target component. In the next step S41, it is determined whether or not an image that has not been selected as a processing target in step S38 is included in the terminal image. If there is an unselected image, the process returns to step S38; otherwise, the process proceeds to step S42. In step S42, it is determined whether there are a plurality of target parts. If there are a plurality of target parts, the process proceeds to step S43; otherwise, the process returns to the operation support process. In step S43, the frequency of appearance in the character string extracted from the terminal image from the display component selected as the target component in step S37 and the display component determined as the target component in step S40, or the terminal image Only the target part corresponding to the candidate part having the highest frequency of occurrence is selected as the display part that is expected to be operated by the user, and the process is returned to the operation support process. A display component corresponding to a candidate portion whose frequency that appears in the character string extracted from the terminal image or frequency that appears in the terminal image is not maximum is excluded from the target components.

  FIG. 10 is a flowchart illustrating an example of the flow of the operation screen change process. The operation screen change process is a process executed in step S07 in FIG. In the prediction process executed in step S06 before the operation screen change process is executed, there are a case where the target part and a candidate portion corresponding to the target part are determined, and a case where they are not determined. Referring to FIG. 10, CPU 111 determines whether a target part has been determined (step S51). If the target part has been determined, the process proceeds to step S52; otherwise, the process proceeds to step S56. In step S56, the terminal image displayed in the second area is reduced and displayed in the second area, and the process returns to the operation support process. Since the terminal image is displayed in a reduced size, it is possible to notify the user that information for assisting the operation does not exist in the terminal image.

  In step S52, the process branches depending on the change mode. The change mode includes a superposition mode, an arrangement change mode, and an instruction mode, and is set to any one in advance. If the change mode is set to the superposition mode, the process proceeds to step S53. If the change mode is set to the arrangement change mode, the process proceeds to step S54. If the change mode is set to the instruction mode, If so, the process proceeds to step S55.

  In step S53, the terminal image displayed in the second area is arranged in the vicinity of the display component determined as the target component, and the process proceeds to step S55. As a result, the terminal image is displayed in the vicinity of the display component determined as the target component, so that the display component to be operated by the user can be shown to the user at the position where the terminal image is arranged.

  In step S54, the arrangement of one or more display components included in the operation screen is changed, and the process proceeds to step S55. The arrangement of one or more display components is changed so that the display component determined as the target component becomes the highest priority position with the highest priority. As a result, the display component determined as the target component is displayed at the highest priority position, so that the display component to be operated by the user can be indicated by the position where the display component is arranged.

  In step S55, the instruction image is displayed on the target part and the candidate part, and the process returns to the operation support process. For example, a line connecting the target part and the candidate portion is displayed on the display unit 161 as an instruction image. Further, the same image is arranged as an instruction image in the vicinity of the target part and in the vicinity of the candidate portion, and is displayed on the display unit 161. For example, a circle surrounding the target part and a circle surrounding the candidate part are displayed on the display unit 161 as instruction images. Thereby, since the instruction image is displayed, it is possible to notify the user that the candidate portion in the terminal image is related to the display component in the operation screen.

  FIG. 11 is a flowchart illustrating an example of the flow of the operation reception process. The operation acceptance process is a process executed in step S09 in FIG. An operation input by the user is received at a stage before the operation receiving process is executed. Referring to FIG. 11, CPU 111 branches the process depending on the type of operation in step S <b> 71. If the operation input by the user is a set value setting operation for setting a set value, the process proceeds to step S72. If the operation input by the user is a screen transition operation for switching the operation screen to another operation screen, the process is performed in step S74. If the operation input by the user is an execution instruction operation for instructing execution of the process, the process proceeds to step S77.

  In step S72, the setting value determined by the setting value setting operation is received, and the process proceeds to step S73. In step S73, the setting value accepted in step S72 is set, and the process returns to the operation support process. When the set value is set, the set value received in step S72 is displayed in the area where the set value is input in the operation screen displayed on the display unit 161.

  In step S74, the operation screen specified by the screen transition operation is determined. The screen transition operation is accepted when, for example, a screen transition button included in the operation screen is instructed by the user. The screen transition button is associated with the operation screen. In this case, the operation screen associated with the screen transition button designated by the user is determined. In step S75, the operation screen determined in step S74 is displayed in the first area of display unit 161, and the process returns to the operation support process.

  In step S76, the process specified by the execution instruction operation is executed according to the set value, and the process is returned to the operation support process.

  FIG. 12 is a first diagram illustrating an example of an image displayed on the MFP. FIG. 12 shows an image displayed in a state before MFP 100 starts communication with smartphone 200. Referring to FIG. 12, display unit 161 includes a first region 351 and a second region 353. An operation screen 311 is displayed in the first area 351, and a notification screen 355 is displayed in the second area 353. The operation screen 311 includes a plurality of transition buttons respectively associated with another plurality of operation screens. For example, a transition button representing “fax” is associated with an operation screen for setting a facsimile number, and a transition button representing “E-mail” is associated with an operation screen for setting an e-mail address.

  The notification screen 355 displayed in the second region 353 is a screen for notifying the user that mobile cooperation is possible, and includes a landmark image 363 arranged in the antenna region 171A. A mark image 363 is a mark indicating the position of the NFC antenna 171. In order to link the smartphone 200 with the MFP 100, the user only needs to perform an operation of bringing the smartphone 200 close to the position where the landmark image 363 is displayed. Therefore, an operation for linking the smartphone 200 with the MFP 100 is facilitated.

  FIG. 13 is a second diagram illustrating an example of an image displayed on the MFP. FIG. 13 shows an image displayed while MFP 100 is communicating with smartphone 200. Referring to FIG. 13, an image displayed after smartphone 200 approaches the position of landmark image 363 in a state where the image shown in FIG. 12 is displayed on display unit 161 is shown. Referring to FIG. 13, operation screen 311 is displayed in first area 351 of display unit 161, and terminal image 421 is displayed in the second area. The operation screen 311 is the same as the image displayed in the first area 351 shown in FIG. The terminal image 421 is the same image as the image displayed on the display unit 206 of the smartphone 200. The terminal image 421 includes a character string “supports the operation of the copier. Please input by voice”.

  FIG. 14 is a third diagram illustrating an example of an image displayed on the MFP. FIG. 14 is a screen that is displayed when the screen displayed on the smartphone 200 changes while the image shown in FIG. 13 is displayed on the display unit 161. Referring to FIG. 14, operation screen 311 is displayed in first area 351 of display unit 161, and terminal image 431 is displayed in the second area. The operation screen 311 is the same as the image displayed in the first area 351 shown in FIG. The terminal image 431 includes a character string “Open box”. In this example, when the user inputs “box” by voice to the smartphone 200 as a word for requesting operation support, the smartphone 200 opens “open box as support information for explaining the operation procedure of the box. Is displayed.

  In addition, in the operation screen displayed in the first area 351, an image 431A obtained by reducing the terminal image is displayed superimposed on the lower side of the button representing the box characters. As a result, the user can know that the button displaying the text of the box is a display component corresponding to the text string “Open the box” included in the terminal image, and perform the operation to open the box. Can be entered.

  FIG. 15 is a fourth diagram illustrating an example of an image displayed on the MFP. FIG. 15 is a screen that is displayed when the screen displayed on the smartphone 200 changes while the image shown in FIG. 13 is displayed on the display unit 161. Referring to FIG. 15, operation screen 311 is displayed in first area 351 of display unit 161, and terminal image 431 is displayed in the second area. The operation screen 311 is the same as the image displayed in the first area 351 shown in FIG. The terminal image 431 includes a character string “Open box”. In this example, when the user inputs “box” by voice to the smartphone 200 as a word for requesting operation support, the smartphone 200 opens “open box as support information for explaining the operation procedure of the box. Is displayed.

  In addition, in the operation screen displayed in the first area 351, the arrangement of the button representing the box character is changed to the position with the highest priority in the upper left. Specifically, in FIG. 13, from left to right, a button representing a fax character, a button representing an E-mail character, and a button representing a box character are arranged in this order. However, in FIG. 15, from left to right, a button representing a box character, a button representing a fax character, and a button representing an E-mail character are arranged in this order.

  As a result, the user can know that the button displaying the text of the box is a display component corresponding to the text string “Open the box” included in the terminal image, and perform the operation to open the box. Can be entered.

  FIG. 16 is a fifth diagram illustrating an example of an image displayed on the MFP. FIG. 16 is a screen that is displayed when the screen displayed on the smartphone 200 changes while the image shown in FIG. 13 is displayed on the display unit 161. Referring to FIG. 15, operation screen 311 is displayed in first area 351 of display unit 161, and terminal image 431 is displayed in the second area. The operation screen 311 is the same as the image displayed in the first area 351 shown in FIG. The terminal image 431 includes a character string “Open box”. In this example, when the user inputs “box” by voice to the smartphone 200 as a word for requesting operation support, the smartphone 200 opens “open box as support information for explaining the operation procedure of the box. Is displayed.

  In addition, a button representing a box character displayed on the operation screen 311 displayed in the first area 351, a character string of “box” included in the terminal image 431 displayed in the second area 353, and , An instruction image 435 is displayed. As a result, the user can know that the button displaying the text of the box is a display component corresponding to the text string “Open the box” included in the terminal image, and perform the operation to open the box. Can be entered.

  FIG. 17 is a sixth diagram illustrating an example of an image displayed on the MFP. FIG. 17 is a screen that is displayed when the screen displayed on the smartphone 200 changes while the image shown in FIG. 13 is displayed on the display unit 161. Referring to FIG. 17, operation screen 311 is displayed in first area 351 of display unit 161, and terminal image 441 is displayed in the second area. The operation screen 311 is the same as the image displayed in the first area 351 shown in FIG. In the terminal image 441, “if the page indicator is displayed, use the page switching key to switch pages.”, “Previous page”, “Next page” character strings, and to switch the page to the previous page Page indicator image 443A and a page indicator image 445A for switching the page to the next page.

  In this example, when the user inputs “page switching” by voice to the smartphone 200 as a word for requesting operation support, the smartphone 200 displays support information for explaining a procedure for operating the box. It is.

  In addition, a page indicator 443 displayed on the operation screen 311 displayed in the first area 351 and a page indicator 445 for switching the page to the next page are displayed. Further, an instruction image 447 surrounding the page indicator 443 of the operation screen 311, an instruction image 447A surrounding the page indicator image 443A included in the terminal image 441, an instruction image 449 surrounding the page indicator 445 of the operation screen 311, and a terminal image An instruction image 449A surrounding a page indicator image 445A included in 441 is displayed. The instruction image 447 and the instruction image 447A have the same shape, and the instruction image 449 and the instruction image 449A have the same shape. The instruction image 447 and the instruction image 449 have different shapes. Thereby, the user can know that the page indicator 443 is a display component corresponding to the page indicator image 443A included in the terminal image, and can input an operation of switching the page to the previous page on the operation screen. it can. Further, the user can know that the page indicator 445 is a display component corresponding to the page indicator image 445A included in the terminal image, and can input an operation of switching the page to the next page on the operation screen. .

  As described above, MFP 100 according to the present embodiment functions as an image forming apparatus, acquires a terminal image displayed on display unit 206 from smartphone 200 functioning as a mobile device, and character information included in the terminal image. And / or the content of the operation screen is changed based on the image. For this reason, the content of the operation screen can be matched with the information included in the terminal image displayed on the smartphone 200. When an application program that supports the operation of MFP 100 by the user is executed on smartphone 200, the content that supports the operation of MFP 100 on smartphone 200 can be displayed on the operation screen of MFP 100. As a result, the operation of MFP 100 can be supported only by the information displayed on MFP 100.

  Further, based on information included in the terminal image, the MFP 100 predicts a display component that is a target operation to be input by the user as a target component, and highlights and displays the predicted display component on the target component. . For this reason, a user can be notified of operation guided with smart phone 200.

  In addition, since MFP 100 predicts a display component that is a target of an operation scheduled to be input by the user based on character information included in the terminal image, the accuracy of predicting the display component can be improved. In particular, since the MFP 100 compares the character string included in the character information with a reserved word associated with the display component in advance, the accuracy of predicting the display component can be improved. Further, since MFP 100 predicts a display component based on a character string that frequently appears from among a plurality of character strings included in character information, the accuracy of predicting the display component can be improved. Further, since MFP 100 predicts a display component similar to the image included in the terminal image as the target component, the accuracy of predicting the display component can be increased.

  Further, when the terminal image includes a plurality of images that are respectively similar to a plurality of display components, MFP 100 predicts a display component that is similar to a high-frequency image among the plurality of images as a target component, and therefore predicts a display component. Accuracy can be increased.

  Further, since the MFP 100 displays the terminal image in the vicinity of the display component for accepting the predicted operation, the MFP 100 can notify the position of the predicted display component on the operation screen. Further, when the operation screen and the terminal image are displayed on the display unit before the target part is predicted, the MFP 100 moves the position where the terminal image is displayed. The predicted display component can be notified. In particular, since the MFP 100 moves the display component predicted as the target component to a position that is prioritized over other display components, the MFP 100 can notify the display component at a priority position on the operation screen.

  Further, since the MFP 100 highlights the display component predicted as the target component and the area where the original information included in the terminal image and from which the display component is predicted is displayed, the display component and the original information are displayed. Can be notified of the relationship. In particular, since an instruction image for associating the display component predicted for the target component with the area where the original information is displayed is displayed, it is possible to notify the user of the display component and the original information related to the display component. it can. If the display component is not predicted as the target component, the terminal image is not displayed, or if the terminal image is displayed, the terminal image is reduced in size, so the terminal image has information related to the display component. You can be notified that you will not.

  In the above-described embodiment, the MFP 100 has been described as an example of an image processing apparatus. However, the operation support method that causes the MFP 100 to execute the operation support process illustrated in FIGS. Needless to say, the present invention can be understood as an operation support program that causes the CPU 111 that controls the MFP 100 to execute the above.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

<Appendix>
(1) The image processing apparatus according to claim 12, wherein the instruction image is a line connecting the display component and a region where the original information is displayed.
(2) The image processing apparatus according to claim 12, wherein the instruction image is the same image arranged in the vicinity of each of the areas where the display component and the original information are displayed.
(3) The image processing apparatus according to any one of claims 1 to 13, wherein the terminal image is stream data.

  1 Information system, 3 networks, 5 radio stations, 100 MFP, 110 main circuit, 111 CPU, 112 communication I / F unit, 113 ROM, 114 RAM, 115 HDD, 116 facsimile unit, 117 external storage device, 118 CD-ROM 119 NFC communication unit, 120 automatic document feeder, 130 document reading unit, 140 image forming unit, 150 paper feeding unit, 155 post-processing unit, 160 operation panel, 161 display unit, 163 operation unit, 165 touch panel, 167 hard key , 171 NFC antenna, 171A antenna area, 173 NFC communication control unit, 175 detection circuit, 200 smartphone, 201 CPU, 202 camera, 203 flash memory, 204 wireless communication unit, 205 call unit, 206 display unit, 207 Cropping unit, 207A main key, 207B touch panel, 209 communication unit, 211 external storage device, 51 image acquisition unit, 53 analysis unit, 55 operation support unit, 57 terminal image display unit, 59 operation reception unit, 61 operation screen display unit, 63 setting value determination unit, 65 process execution unit, 67 character extraction unit, 69 image extraction unit, 71 prediction unit, 73 comparison unit, 75 highlight display unit, 81 neighborhood arrangement unit, 83 arrangement change unit, 85 related display unit, 87 Reduction unit, 311 operation screen, 351 first area, 353 second area, 421, 431, 441 terminal image, 443, 445 page indicator, 435, 447, 447A, 449, 449A instruction image.

Claims (15)

An image processing apparatus capable of communicating with a portable device,
Display means for displaying an operation screen;
Image obtaining means for obtaining a terminal image displayed by the portable device from the portable device;
An image processing apparatus comprising: an operation support unit that changes the content of the operation screen based on information included in the acquired terminal image. The operation screen includes a display component for receiving an operation input by a user,
The operation support means, based on information included in the terminal image, a prediction means for predicting the display component that is a target of the operation scheduled to be input by a user as a target component;
The image processing apparatus according to claim 1, further comprising: highlighting means for highlighting and displaying the predicted target part.   The image processing apparatus according to claim 2, wherein the predicting unit predicts the display component based on character information included in the terminal image.   The image processing apparatus according to claim 3, wherein the predicting unit compares a character string included in the character information with a reserved word associated with the display component in advance.   The image processing apparatus according to claim 4, wherein the predicting unit predicts the display component based on a character string that appears frequently from a plurality of character strings included in the character information.   The image processing apparatus according to claim 2, wherein the prediction unit predicts the display component similar to an image included in the terminal image as the target component.   When the terminal image includes a plurality of the images similar to the plurality of display components, the prediction means predicts the display component similar to the frequently used image among the plurality of images as the target component. The image processing apparatus according to claim 6.   The image processing apparatus according to claim 2, wherein the highlighting unit displays the terminal image in the vicinity of the display component for receiving the predicted operation.   The highlighting means moves a position where the terminal image is displayed when the operation screen and the terminal image are displayed on the display means before the target part is predicted. An image processing apparatus according to 1.   The image processing apparatus according to claim 2, wherein the highlighting unit moves the display component predicted for the target component to a position that is prioritized over the other display components. The display means displays the terminal image in addition to the operation screen,
The highlighting means highlights the display component predicted for the target component and a region in which the original information included in the terminal image and from which the target component is predicted is displayed. The image processing apparatus according to any one of 2 to 7.   The image processing apparatus according to claim 11, wherein the highlighting unit displays an instruction image associating the display component predicted for the target component with an area where the original information is displayed on the display unit.   The highlighting means does not display the terminal image when the display component is not predicted as the target component, or reduces the size of the terminal image when the terminal image is displayed. The image processing apparatus in any one of -12. An image obtaining step for obtaining a terminal image displayed by the portable device from a portable device;
An operation support method for causing an image processing apparatus to execute an operation support step of changing a content of an operation screen displayed on a display unit based on information included in the acquired terminal image. An operation support program executed by a computer that controls the image processing apparatus,
An image obtaining step for obtaining a terminal image displayed by the portable device from a portable device;
An operation support program for causing the computer to execute an operation support step of changing a content of an operation screen displayed on a display unit based on information included in the acquired terminal image.

Images