JP2015207913A - Remote control system, image processing device, remote control method, and remote control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display, on a remote operation device, a remote operation screen including an operation screen whose size is made to be large and a hard key screen whose size is made to be as small as possible.SOLUTION: An MFP comprises: an operation acceptance unit comprising a plurality of hard keys; a display unit for displaying an operation screen; an identification unit 55 for identifying an application program corresponding to the operation screen displayed on the display unit; and a use hard key determination unit 61 for determining that one or more hard keys prescribed by the application program identified by the identification unit 55 are taken as use hard keys. Any of the MFP and a portable information device creates, on the basis of hard key information on the one or more use hard keys, a hard key screen in which one or more soft key images individually corresponding to the one or more hard keys are arranged; and creates a remote operation screen including the operation screen and the created hard key screen.

Description

  The present invention relates to a remote control system, an image processing apparatus, a remote control method, and a remote control program, and in particular, is executed by a remote control system for remotely operating an image processing apparatus from a remote operation apparatus, the image processing apparatus, and the image processing apparatus. The present invention relates to a remote control method, a remote control method executed by the image processing apparatus, and a remote control program.

  2. Description of the Related Art In recent years, a technology for remotely operating a multifunction peripheral (hereinafter referred to as “MFP”) using a portable information device such as a smartphone or a tablet terminal is known. An operation screen for operating the MFP can be displayed on the portable information device, and the MFP can be remotely operated by the portable information device. For example, in JP 2012-93940 A, the operated terminal receives from the mobile terminal a display screen control unit that creates a mobile terminal screen in which the operated terminal screen is modified based on the display attribute information of the mobile terminal. The mobile terminal includes an encoding unit that encodes the mobile terminal screen using the encoding method and an operation processing unit that executes an operation specified by position information received from the mobile terminal. When a decoding unit that decodes the terminal screen and outputs it to the touch screen and a touch operation on the touch screen displaying the mobile terminal screen is detected, position information indicating the touch position on the mobile terminal screen is displayed as its own display attribute information. And a position information correction unit for correcting the position information on the operated terminal screen based on the display attribute information of the operated terminal received from the operated terminal. There has been described.

However, the conventional MFP has a hard key including a contact switch such as a start key, whereas the portable information device has a hard key, but the number is larger than the number of hard keys of the MFP. Few. For this reason, there is a problem in that the portable information device may not be able to accept all operations of pressing hard keys input to the MFP.
JP 2012-93940 A

  The present invention has been made to solve the above-described problems, and one of the objects of the present invention is to provide a remote operation device having a remote operation screen in which the size of the operation screen is increased by reducing the size of the hard key screen as much as possible. It is an object of the present invention to provide a remote control system that can be displayed on the screen.

  Another object of the present invention is to provide an image processing apparatus capable of displaying on a remote operation device a remote operation screen in which the size of the operation screen is increased by reducing the size of the hard key screen as much as possible.

  Still another object of the present invention is to provide a remote control method capable of displaying on a remote operation device a remote operation screen in which the size of the operation screen is increased by reducing the size of the hard key screen as much as possible.

  Still another object of the present invention is to provide a remote control program capable of causing a remote operation device to display a remote operation screen in which the size of the operation screen is increased by reducing the size of the hard key screen as much as possible.

  In order to achieve the above object, according to one aspect of the present invention, a remote control system is a remote control system including an image processing device and a remote operation device for remotely operating the image processing device, An apparatus includes an operation accepting means having a plurality of hard keys, a display means for displaying an operation screen, a specifying means for specifying an application program corresponding to an operation screen displayed on the display means from among a plurality of application programs, Use hard key determining means for determining one or more hard keys defined by the application program specified by the specifying means among the plurality of hard keys as the use hard keys, and any of the image processing apparatus and the remote control apparatus. Is determined based on hard key information about the determined one or more used hard keys. A remote operation screen for generating a remote operation screen including a hard key screen generating means for generating a hard key screen on which one or more soft key images corresponding to each hard key are arranged, and an operation screen and the generated hard key screen Generating means.

  According to this aspect, the application program corresponding to the operation screen displayed on the display unit is identified from the plurality of application programs by the image processing apparatus, and is determined by the identified application program among the plurality of hard keys. One or more hard keys are determined to be used hard keys, and a hard key screen is generated based on the hard key information related to the determined one or more used hard keys by either the image processing device or the remote control device. And a remote operation screen including a hard key screen is generated. For this reason, a remote operation screen capable of designating a hard key defined by an application program corresponding to the operation screen displayed on the image processing device among the plurality of hard keys provided in the image processing device is displayed on the remote operation device. be able to. As a result, it is possible to provide a remote control system capable of displaying on the remote operation device a remote operation screen in which the size of the hard key screen is reduced as much as possible and the size of the operation screen is increased.

  Preferably, the used hard key determining means includes at least one of the plurality of hard keys that can be accepted by the operation accepting means while the application program specified by one or more hard keys excluding the power key is being executed. Application unit determining means for determining a hard key as a use hard key is included.

  According to this aspect, the hard key screen can be fixed while the same application program is being executed in the image processing apparatus.

  Preferably, the used hard key determining means includes one or more hard keys that can be received by the operation receiving means while the operation screen is displayed on the display means among one or more hard keys excluding the power key among the plurality of hard keys. Screen unit determining means for determining a hard key as a used hard key is included.

  According to this aspect, since one or more hard keys that can be accepted while the operation screen is displayed are determined as the use hard keys, the size of the hard key screen can be reduced as much as possible.

  Preferably, the image processing apparatus executes the application program, transmits the operation screen to the remote operation device, and receives the operation screen from the remote operation device among the plurality of application programs. Remote control means for causing the process execution means to execute an application program specified by the remote operation.

  According to this aspect, the application program can be executed by remotely operating the image processing device from the remote operation device, and the same operation screen as that displayed on the image processing device can be displayed on the remote operation device. it can.

  Preferably, the used hard key determining means includes a program analyzing means for analyzing an application program executed by the process executing means.

  Preferably, the used hard key determining means includes language analyzing means for analyzing the operation screen when the operation screen generated by the process execution means is described in a markup language.

  Preferably, the image processing apparatus further includes server remote operation means for displaying an operation screen received from an external server on the display means, and transmitting the remote operation accepted by the operation accepting means to the server. The server remote control means specifies an application program to be executed by an external server.

  According to this aspect, when the server is remotely operated from the image processing apparatus, the image processing apparatus and the server can be remotely operated from the remote operation apparatus.

  Preferably, the used hard key determining means includes hard key information acquiring means for acquiring information on the hard key from an external server.

  According to this aspect, since information on hard keys is acquired from an external server, a hard key screen including hard keys necessary for remotely operating the server can be displayed.

  Preferably, when the application program specified by the specifying unit is a predetermined specific program, the used hard key determining unit determines all of the plurality of hard keys except the power key as used hard keys.

  According to another aspect of the present invention, an image processing device is an image processing device that is remotely operated by a remote operation device, the operation receiving means having a plurality of hard keys, and an operation described in a markup language. Display means for displaying a screen, use hard key determination means for determining one or more use hard keys used on the operation screen among a plurality of hard keys by analyzing the displayed operation screen, and determined 1 Hard key screen generating means for generating a hard key screen in which one or more soft key images corresponding to each of one or more used hard keys are arranged in a markup language based on the hard key information relating to the above used hard keys, and an operation A remote operation screen generating means for generating a remote operation screen including the screen and the generated hard key screen; and Transmitted to, provided in accordance with the remote operation received from the remote operation device, a remote control means that executes processing, the.

  According to this aspect, by analyzing the operation screen described in the markup language, one or more used hard keys to be used on the operation screen are determined from among the plurality of hard keys, and the determined one or more used hardware is determined. A hard key screen with one or more soft key images corresponding to each key is generated in a markup language, a remote operation screen including an operation screen and a hard key screen is generated, and the remote operation screen is transmitted to the remote operation device. Is done. Therefore, it is possible to provide an image processing apparatus capable of displaying a remote operation screen with a hard key screen size as small as possible and a large operation screen size on the remote operation device.

  Preferably, the apparatus further comprises processing execution means for executing an application program and generating an operation screen for display on the display means in a markup language, and the remote control means is received from a remote operation device among the plurality of application programs. The process execution means is caused to execute an application program specified by remote operation.

  According to this aspect, the application program is executed under the control of the remote operation device, and the same operation screen as the operation screen displayed as a result can be displayed on the remote operation device.

  Preferably, the process execution means further includes server remote operation means for receiving an operation screen described in a markup language received from an external server and transmitting the server remote operation received by the operation reception means to the server. When the operation screen is received by the server remote operation means, the remote control means transmits the remote operation screen including the received operation screen to the remote operation device, and then receives the remote operation received from the remote operation device at the server remote As an operation, it is transmitted to the server remote control means.

  According to this aspect, when the server is remotely operated from the image processing apparatus, the image processing apparatus and the server can be remotely operated by the remote operation apparatus.

  Preferably, the used hard key determining means includes hard key information acquiring means for acquiring information on the hard key from an external server.

  According to this aspect, since information on hard keys is acquired from an external server, a hard key screen including hard keys necessary for remotely operating the server can be displayed.

  Preferably, the used hard key information acquiring unit includes an app unit acquiring unit that receives information on the hard key before receiving an operation screen from an external server.

  According to this aspect, the hard key screen can be fixed while the same application program is being executed on the server.

  Preferably, the used hard key determining unit includes a screen unit acquiring unit that receives information on the hard key every time an operation screen is received from an external server.

  According to this aspect, every time an operation screen is received from the server, the hard key to be used is determined, so that the size of the hard key screen can be made as small as possible.

  According to still another aspect of the present invention, a remote control method is a remote control method executed by an image processing device and a remote operation device for remotely operating the image processing device, and the image processing device includes a plurality of remote control methods. An operation receiving means having a hard key and a display means for displaying an operation screen are provided, and an application program corresponding to the operation screen displayed on the display means is identified from among a plurality of application programs in the image processing apparatus. And a use hard key determination step of determining one or more hard keys defined by the application program specified in the specific step among the plurality of hard keys as a use hard key, Hard key information on one or more hard keys determined to be used on any of the remote control devices Accordingly, a remote operation screen including a hard key screen generation step for generating a hard key screen in which one or more soft key images corresponding to one or more used hard keys are arranged, and an operation screen and the generated hard key screen. And a remote operation screen generating step for generating.

  According to this aspect, it is possible to provide a remote control method capable of displaying on the remote operation device a remote operation screen in which the size of the operation screen is increased by reducing the size of the hard key screen as much as possible.

  According to still another aspect of the present invention, a remote control method is a remote control method executed by an image processing device remotely operated by a remote operation device, and the image processing device includes a plurality of hard keys. Operation receiving means and display means for displaying an operation screen described in a markup language are provided, and one or more used on the operation screen among a plurality of hard keys by analyzing the displayed operation screen One or more soft key images corresponding to each of the one or more used hard keys are arranged based on the used hard key determining step for determining the used hard key and the hard key information related to the determined one or more used hard keys. A remote operation screen including a hard key screen generation step for generating a hard key screen in a markup language, and an operation screen and the generated hard key screen A remote operation screen generating step of generating is, the generated remote control screen and transmitted to the remote operation apparatus, according to the remote operation received from the remote operation device, a remote control step of executing a process, thereby executing.

  According to this aspect, it is possible to provide a remote control method capable of displaying on the remote operation device a remote operation screen in which the size of the operation screen is increased by reducing the size of the hard key screen as much as possible.

  According to still another aspect of the present invention, the remote control program is a remote control program that is executed by a computer that controls an image processing device that is remotely operated by a remote operation device, and the image processing device includes a plurality of hardware components. Operation receiving means with a key and display means for displaying an operation screen described in a markup language are provided and used on the operation screen among a plurality of hard keys by analyzing the displayed operation screen One or more soft keys corresponding to each of the one or more used hard keys based on the hard key information relating to the one or more used hard keys and the determined one or more used hard keys A hard key screen generation step for generating a hard key screen in which an image is arranged in a markup language, and an operation screen are generated. A remote control screen generating step for generating a remote control screen including a hard key screen, and a remote control for transmitting the generated remote control screen to the remote control device and executing processing according to the remote control received from the remote control device Steps are executed by a computer.

  According to this aspect, it is possible to provide a remote control program capable of displaying on the remote operation device a remote operation screen in which the size of the operation screen is increased by reducing the size of the hard key screen as much as possible.

It is a figure showing the whole remote control system outline in one of the embodiments of the present invention. 1 is a perspective view showing an appearance of an MFP. 2 is a block diagram illustrating an outline of a hardware configuration of an MFP. FIG. It is a top view which shows an example of an operation panel. It is a block diagram which shows the outline | summary of the hardware constitutions of a portable information device. 3 is a block diagram illustrating an example of functions of a CPU provided in the MFP according to the first embodiment. FIG. It is a block diagram which shows an example of the outline | summary of the function of CPU with which the portable information device in 1st Embodiment is provided. It is a 1st figure which shows an example of an operation screen. It is a 1st figure which shows an example of a remote operation screen. It is a 2nd figure which shows an example of a remote operation screen. It is a 2nd figure which shows an example of an operation screen. It is a 3rd figure which shows an example of a remote operation screen. It is a 3rd figure which shows an example of an operation screen. It is a 4th figure which shows an example of a remote operation screen. It is a 4th figure which shows an example of an operation screen. It is a 5th figure which shows an example of a remote operation screen. It is a flowchart which shows an example of the flow of the remote operation processing in 1st Embodiment. It is a flowchart which shows an example of the flow of a screen production | generation process. It is a flowchart which shows an example of the flow of the remote control process in 1st Embodiment. FIG. 10 is a block diagram illustrating an example of functions of a CPU provided in an MFP according to a second embodiment. It is a block diagram which shows an example of the detailed function of 63 A of remote control reception parts. It is a block diagram which shows an example of the outline | summary of the function of CPU with which the portable information device in 2nd Embodiment is provided. It is a flowchart which shows an example of the flow of the remote operation process in 2nd Embodiment. It is a flowchart which shows an example of the flow of the remote control process in 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

<First Embodiment>
FIG. 1 is a diagram showing an overall outline of a remote control system according to one embodiment of the present invention. Referring to FIG. 1, a remote control system 1 includes a multifunction peripheral (hereinafter referred to as “MFP”) 100 that functions as an image processing apparatus, a personal computer (hereinafter referred to as “PC”) 300, a wireless station 5, Mobile information devices 200, 200A, and 200B. An MFP (Multi Function Peripheral) 100 and a PC 300 are connected to the network 3. The portable information devices 200, 200 </ b> A, 200 </ b> B are connected to the network 3 through the wireless station 5.

  The network 3 is a local area network (LAN), and the connection form may be wired or wireless. The network 3 is not limited to a LAN, and may be a wide area network (WAN), a public switched telephone network (PSTN), the Internet, or the like.

  The PC 300 is a general computer. The PC 300 is installed with a printer driver program corresponding to the MFP 100, and by executing the printer driver program, the MFP 100 can be controlled to cause the MFP 100 to execute image forming processing, document reading processing, and the like. .

  Portable information devices 200, 200A, and 200B are computers that are carried and used by users, such as smartphones and PDAs (Personal Digital Assistants). Since the portable information devices 200, 200A, and 200B have the same hardware configuration and functions, the portable information device 200 will be described as an example unless otherwise specified. Here, the portable information device 200 is a smartphone, and has a wireless LAN function and a call function. For this reason, the portable information device 200 is connected to the cellular phone network by wirelessly communicating with the cellular phone base station, and can make a call.

  In remote control system 1 according to the present embodiment, portable information devices 200, 200A, and 200B can remotely control MFP 100. In this case, each of portable information devices 200, 200A, and 200B functions as a remote operation device that remotely operates MFP 100, and MFP 100 functions as a remote control device that is remotely operated by each of portable information devices 200, 200A, and 200B. Here, a case where portable information device 200 remotely controls MFP 100 will be described as an example. In this case, a remote control program that is remotely controlled by portable information device 200 is installed in MFP 100, and a remote operation program for remotely controlling MFP 100 is installed in portable information device 200. The remote operation program may be a dedicated application program developed corresponding to MFP 100, or may be a browsing program when MFP 100 functions as a Web server.

  Portable information device 200 receives and displays a remote operation screen for remote operation from MFP 100. When a user who operates portable information device 200 inputs a remote operation according to the remote operation screen, portable information device 200 transmits the remote operation to MFP 100. On the other hand, when MFP 100 transmits a remote operation screen to portable information device 200 and receives a remote operation from portable information device 200, MFP 100 executes processing according to the received remote operation.

  In the present embodiment, a case where a remote operation program for remotely controlling MFP 100 is installed in portable information device 200 will be described as an example. However, a remote operation program for remotely controlling MFP 100 is installed in PC 300. May be. In this case, PC 300 serves as a remote operation device, and the user can remotely control MFP 100 using PC 300 as in the case of remotely controlling MFP 100 using portable information device 200.

  FIG. 2 is a perspective view showing the appearance of the MFP. FIG. 3 is a block diagram showing an outline of the hardware configuration of the MFP. 2 and 3, MFP 100 includes a main circuit 110, a document reading unit 130 for reading a document, an automatic document transport device 120 for transporting a document to document reading unit 130, and a document reading unit. An image is formed by an image forming unit 140 for forming an image on a sheet or the like based on image data output by the manuscript 130 reading and outputting a document, a paper feeding unit 150 for supplying the image to the image forming unit 140, and an image. A post-processing unit 155 that processes paper and an operation panel 160 as a user interface are included.

  The post-processing unit 155 executes a sorting process for sorting and discharging one or more sheets on which images are formed by the image forming unit 140, a punching process for punch hole processing, and a staple process for driving staple needles.

  The main circuit 110 includes a CPU 111, a communication interface (I / F) unit 112, a ROM 113, a RAM 114, a hard disk drive (HDD) 115 as a mass storage device, a facsimile unit 116, and a CD-ROM 118. External storage device 117. CPU 111 is connected to automatic document feeder 120, document reading unit 130, image forming unit 140, paper feeding unit 150, post-processing unit 155, and operation panel 160, and controls the entire MFP 100.

  The ROM 113 stores a program executed by the CPU 111 or data necessary for executing the program. The RAM 114 is used as a work area when the CPU 111 executes a program. Further, the RAM 114 includes a setting value storage area and a setting screen storage area. The set value storage area is an area for storing a set value for executing processing. The setting screen storage area is an area for storing a setting screen displayed on the display unit 161. The RAM 114 temporarily stores read data (image data) continuously sent from the document reading unit 130.

  Communication I / F unit 112 is an interface for connecting MFP 100 to network 3. The CPU 111 communicates with the PC 300 or the portable information devices 200, 200A, and 200B via the communication I / F unit 112 to transmit and receive data. The communication I / F unit 112 can communicate with a computer connected to the Internet via the network 3.

  The facsimile unit 116 is connected to the public switched telephone network (PSTN) and transmits facsimile data to the PSTN or receives facsimile data from the PSTN. The facsimile unit 116 stores the received facsimile data in the HDD 115 or outputs it to the image forming unit 140. The image forming unit 140 prints the facsimile data received by the facsimile unit 116 on a sheet. Further, the facsimile unit 116 converts the data stored in the HDD 115 into facsimile data, and transmits the facsimile data to a facsimile machine connected to the PSTN.

  The external storage device 117 is loaded with a CD-ROM (Compact Disk ROM) 118. The CPU 111 can access the CD-ROM 118 via the external storage device 117. The CPU 111 loads the program recorded on the CD-ROM 118 attached to the external storage device 117 to the RAM 114 and executes it. The medium for storing the program executed by the CPU 111 is not limited to the CD-ROM 118, but an optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC card, optical card, It may be a semiconductor memory such as a mask ROM, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM).

  Further, the program executed by the CPU 111 is not limited to the program recorded on the CD-ROM 118, and the program stored in the HDD 115 may be loaded into the RAM 114 and executed. In this case, another computer connected to the network 3 may rewrite the program stored in the HDD 115 of the MFP 100 or may add a new program and write it. Further, MFP 100 may download a program from another computer connected to network 3 and store the program in HDD 115. The program here includes not only a program directly executable by the CPU 111 but also a source program, a compressed program, an encrypted program, and the like.

  Operation panel 160 is provided on the upper surface of MFP 100. FIG. 4 is a plan view showing an example of the operation panel. Note that the dotted line in the figure is attached to indicate a hard key portion 167 described later, and does not actually exist. Referring to FIG. 4, operation panel 160 includes a display unit 161, a touch panel 165, and a hard key unit 167. When the user operates the MFP 100, the user often takes an upright posture, and thus the display surface of the display unit 161, the operation surface of the touch panel 165, and the hard key unit 167 are arranged facing upward. This is because the user can easily visually recognize the display surface of the display unit 161 and the user can easily instruct the operation unit 163 with a finger.

  The display unit 161 is, for example, a liquid crystal display device (LCD), and displays an instruction menu for a user, information about acquired image data, and the like. The touch panel 165 is provided to overlap the display unit 161 on the upper surface or the lower surface of the display unit 161. The touch panel 165 detects a position designated by the user on the display surface of the display unit 161. Hard key portion 167 includes a plurality of hard keys. The hard key is, for example, a contact switch.

  Hard key portion 167 includes a plurality of hard keys classified into first group 410, second group 420, third group 430, and fourth group 440, and hard key 451 for switching power on and off. . In addition, the dotted line in a figure is attached | subjected in order to show the 1st-4th group, and does not actually exist.

  The first group 410 includes five hard keys 411 to 415 having an interrupt processing function that allows the user to execute another process while operating the MFP 100. In the second group 420, twelve numeric keys 422 and hard keys 421, 426, and 427 having an input processing function for inputting set values are classified.

  In the third group 430, three hard keys 431 to 433 having a job control function to which a command for controlling a job executed by the MFP 100 is assigned are classified. The fourth group 440 includes three hard keys 441 to 443 having a shortcut function to which a command for switching to a predetermined operation screen is assigned.

  FIG. 5 is a block diagram showing an outline of the hardware configuration of the portable information device. Referring to FIG. 5, portable information device 200 in the present embodiment includes a CPU 201 for controlling the entire portable information device 200, a camera 202, a flash memory 203 for storing data in a nonvolatile manner, and a calling unit. 205 includes a wireless communication unit 204 connected to 205, a display unit 206 that displays information, an operation unit 207 that receives user operations, a wireless LAN I / F 208, an acceleration sensor 210, and an external storage device 211.

  The display unit 206 is a display device such as a liquid crystal display (LCD) or an organic ELD, and displays an instruction menu for a user, information about acquired image data, and the like. The operation unit 207 includes a main key 207A and a touch panel 207B. When the user instructs the display surface of the display unit 206, the operation unit 207 outputs the position of the display surface detected by the touch panel 207B to the CPU 201. Based on the position detected by the touch panel 207B, the CPU 201 detects a position designated by the user in the screen displayed on the display unit 206. Based on the screen displayed on the display unit 206 and the position detected by the touch panel 207 </ b> B, the CPU 201 accepts input of various instructions, data such as characters and numbers by user operations. For example, when a screen including a numeric keypad image is displayed on the display unit 206, a number corresponding to the key displayed at the position detected by the touch panel 207B is received.

  The camera 202 includes a lens and a photoelectric conversion element. The light focused by the lens is imaged on the photoelectric conversion element. The photoelectric conversion element photoelectrically converts the received light and outputs image data to the CPU 201. The photoelectric conversion element is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like.

  The wireless communication unit 204 performs wireless communication with a mobile phone base station connected to a telephone communication network. The wireless communication unit 204 connects the portable information device 200 to a telephone communication network and enables a call using the call unit 205. The radio communication unit 204 decodes a voice signal obtained by demodulating the radio signal received from the mobile phone base station and outputs the decoded signal to the call unit 205. Further, the wireless communication unit 204 encodes the voice input from the call unit 205 and transmits it to the mobile phone base station. The call unit 205 includes a microphone and a speaker, and outputs the sound input from the wireless communication unit 204 from the speaker, and outputs the sound input from the microphone to the wireless communication unit 204. Further, the wireless communication unit 204 is controlled by the CPU 201, connects the portable information device 200 to an electronic mail server, and transmits and receives electronic mail.

  The wireless LAN I / F 208 is an interface for communicating with the wireless station 5 and connecting the portable information device 200 to the network 3. By registering the IP (Internet Protocol) addresses of PC 300 and MFP 100 in portable information device 200, portable information device 200 can communicate with PC 300 and MFP 100, and data can be transmitted and received. In the present embodiment, a case where portable information device 200 communicates with PC 300 and MFP 100 using wireless LAN I / F 208 will be described as an example, but communication may be performed using another communication method. . Specifically, when portable information device 200, PC 300, and MFP 100 are equipped with a short-range wireless device such as Bluetooth (registered trademark), portable information device 200 has one-to-one correspondence with either PC 300 or MFP 100. You may make it communicate by.

  The flash memory 203 stores a program executed by the CPU 201 or data necessary for executing the program. The CPU 201 loads the program recorded in the flash memory 203 to the RAM included in the CPU 201 and executes it.

  The acceleration sensor 210 is a triaxial acceleration sensor that measures accelerations in three directions of the X axis, the Y axis, and the Z axis. The acceleration sensor 210 outputs the detected acceleration to the CPU 201.

  The external storage device 211 is detachable from the portable information device 200, and a CD-ROM 210A storing a remote operation program can be attached thereto. The CPU 201 can access the CD-ROM 210 </ b> A via the external storage device 211. The CPU 201 can load a remote operation program recorded on the CD-ROM 210A attached to the external storage device 211 into a RAM included in the CPU 201 and execute it.

  The program recorded in the flash memory 203 or the CD-ROM 210A has been described as a program executed by the CPU 201. However, a program in which another computer connected to the network 3 rewrites the program stored in the flash memory 203, Alternatively, it may be a new program written additionally. Further, the portable information device 200 may be a program downloaded from another computer connected to the network 3. The program here includes not only a program directly executable by the CPU 201 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 201 is not limited to the CD-ROM 211A, and may be a semiconductor memory such as an optical disk (MO / MD / DVD), IC card, optical card, mask ROM, EPROM, or EEPROM. Good.

  FIG. 6 is a block diagram illustrating an example of functions of the CPU provided in the MFP according to the first embodiment. The functions shown in FIG. 6 are functions formed in CPU 111 when CPU 111 provided in MFP 100 executes a remote control program stored in ROM 113, HDD 115, and CD-ROM 118. Referring to FIG. 6, CPU 111 includes a remote control unit 51, a process execution unit 53, a specification unit 55, a display control unit 57, an operation reception unit 59, and a used hard key determination unit 61.

  Remote control unit 51 controls communication I / F unit 112 to establish a communication path with the remote operation device, and sets the remote operation device to the user interface of MFP 100. Here, a portable information device 200 will be described as an example of a remote operation device. In this case, the remote control unit 51 sets the portable information device 200 as a user interface. The remote control unit 51 transmits an operation screen to the portable information device 200 according to a request from the portable information device 200 and receives a remote operation from the portable information device 200.

  The remote control unit 51 is a task for the CPU 111 to execute a remote control program that is an application program for remote control. The remote control program communicates with an IWS (Internal Web Server) application program for communicating with portable information device 200 that executes a browsing program and with portable information device 200 that executes a dedicated communication program for communicating with MFP 100. And a dedicated program. In the case where the CPU 111 is a task for executing the IWS application program, the remote control unit 51 communicates with the portable information device 200 using the HTTP protocol. When the CPU 111 is a task for executing a dedicated program, the remote control unit 51 communicates with the portable information device 200 using a predetermined protocol.

  Remote control unit 51 includes a remote operation reception unit 63, an operation screen transmission unit 65, and a hard key information transmission unit 67. The operation screen transmission unit 65 transmits the operation screen input from the process execution unit 53 to the portable information device 200. The remote operation receiving unit 63 receives a remote operation from the portable information device 200 and outputs the received remote operation to the process execution unit 53.

  The hard key information transmission unit 67 receives hard key identification information from a hard key information acquisition unit 85 described later. The hard key identification information is information for identifying each of a plurality of hard keys included in the hard key unit 167. When a communication path is established with the portable information device 200 that is a remote control device, the hard key information transmission unit 67 receives the hard key information including the input hard key identification information via the communication I / F unit 112. Transmit to portable information device 200.

  The operation receiving unit 59 and the display control unit 57 respectively control the operation unit 163 and the display unit 161 included in the operation panel 160 that is a user interface. The operation reception unit 59 controls the operation unit 163, receives an operation input to the operation unit 163 by the user, and outputs the received operation to the process execution unit 53. The display control unit 57 controls the display unit 161 to display the operation screen input from the process execution unit 53 on the display unit 161.

  The process execution unit 53 is a task for the CPU 111 to execute an application program, and executes a process according to an operation that a user inputs according to an operation screen. The process execution unit 53 displays an operation screen on the display unit 161 and accepts an operation input by the user to the operation unit 163. When the remote control unit 51 sets the portable information device 200 to the user interface of the MFP 100, the process execution unit 53 displays an operation screen on the portable information device 200, and the user accepts an input operation to the portable information device 200. When the portable information device 200 is set as the user interface, the operation screen is displayed on the display unit 161 and also displayed on the portable information device 200. When the portable information device 200 is not set in the user interface, the operation screen is displayed on the display unit 161 but is not displayed on the portable information device 200.

  The process execution unit 53 includes a server remote operation unit 71 and a single execution unit 73. In FIG. 6, the flow of information when the server remote operation unit 71 executes processing is indicated by a thick arrow, and the information flow when the operation panel 160 is used as a user interface when the single execution unit 73 executes processing. Is indicated by a thin line arrow.

  The server remote operation unit 71 is a task for executing a server remote operation program, which is an application program for the CPU 111 to remotely operate a server to be remotely operated. The server remote operation unit 71 remotely operates a server to be remotely operated. A server that is a target of remote operation executes an application program in order to provide a predetermined service. Hereinafter, an application program executed by a server to be remotely operated is referred to as a server program. The server program corresponds to the MFP 100 and includes a description of an API (Application Program Interface) command that has been published to enable the MFP 100 to be controlled. “Public” means a state in which a third party other than the manufacturer that manufactures the MFP 100 can use it. For example, the administrator of MFP 100 registers a server and a server program to be executed by the server in MFP 100, whereby the server program to be remotely operated in MFP 100 can be executed by the server.

  Server remote operation unit 71 includes a server remote operation screen reception unit 77 and a server remote operation transmission unit 75. The server remote operation screen receiving unit 77 receives a server remote operation screen transmitted from the server. The server remote operation screen reception unit 77 outputs the received server remote operation screen to the display control unit 57 and the operation screen transmission unit 65. The display control unit 57 displays the server remote operation screen as an operation screen on the display unit 161, and the operation screen transmission unit 65 transmits the server remote operation screen as an operation screen to the portable information device 200 that is a remote operation device. .

  The server remote operation transmission unit 75 transmits an operation input by the user to the server as a server remote operation. The operation input by the user includes an operation for instructing an arbitrary position on the operation screen (server operation screen) and an operation for instructing any of a plurality of hard keys provided in the hard key unit 167. When a remote operation is input from the remote operation receiving unit 63, the server remote operation transmission unit 75 transmits the remote operation to the server as a server remote operation. The remote operation includes a remote operation indicating a position in the operation screen and a remote operation including hard key identification information.

  When an operation is input from the operation reception unit 59, the server remote operation transmission unit 75 transmits the operation as a server remote operation to the server. When the position information indicating the position detected by the touch panel 165 is input to the operation reception unit 59, the server remote operation determined on the server remote operation screen corresponding to the position on the operation screen is transmitted to the server. When the operation reception unit 59 detects pressing of any of the plurality of hard keys included in the hard key unit 167, the server remote operation determined on the server remote operation screen is transmitted to the server corresponding to the pressed hard key. To do.

  The server remote operation screen receiving unit 77 further outputs an acquisition instruction to the hard key information acquisition unit 85 in response to receiving the server remote operation screen transmitted from the server. The acquisition instruction includes server identification information for identifying the server. The server identification information is, for example, a network address assigned to the server.

  The single execution unit 73 is a task for the CPU 111 to execute an application program for controlling the MFP 100. The single execution unit 73 controls the hardware provided in the MFP 100 according to the operation by the user, and executes the process defined by the application program. The hardware included in the MFP 100 includes, for example, an automatic conveyance device 120, a document reading unit 130, an image forming unit 140, a paper feeding unit 150, a post-processing unit 155, a communication I / F unit 112, a facsimile unit 116, an HDD 115, and an external storage device. 117 is included.

  Application programs for controlling MFP 100 executed by CPU 111 include a native program developed exclusively for MFP 100, an IWS program that causes CPU 111 to function as a Web server, and an API program that includes a description of API commands. .

  When the single execution unit 73 is a task for executing a native program, the operation panel 160 is set as a user interface. When the single execution unit 73 is a task for executing an API program, the operation panel 160 is set as a user interface. The display control unit 57 displays on the display unit 161 an operation screen in an image data format output by a task that the CPU 111 executes a native program or a task that the CPU 111 executes an API program.

  When the single execution unit 73 is a task for executing the IWS program, the operation panel 160 is set as the user interface. In this case, the display control unit 57 is a task for the CPU 111 to execute the browsing program, and displays an operation screen described in the markup language output by the single execution unit 73 that is a task for executing the IWS program on the display unit 161. To do.

  The single execution unit 73 executes processing according to an operation input by the user. The single execution unit 73 executes processing according to the remote operation when a remote operation is input from the remote operation reception unit 63, and executes processing according to the operation when the operation is input from the operation reception unit 59.

  The operation input by the user includes an operation for instructing an arbitrary position in the operation screen and an operation for instructing any of a plurality of hard keys provided in the hard key unit 167. When the remote operation is input from the remote operation receiving unit 63, the single execution unit 73, if the remote operation includes position information indicating the position on the operation screen, corresponds to the position on the operation screen by the application program. If the predetermined process is executed and the remote operation includes hard key identification information, the process determined by the application program is executed corresponding to the hard key specified by the hard key identification information. When the operation receiving unit 59 receives position information indicating the position detected by the touch panel 165, the single execution unit 73 executes processing determined by the application program corresponding to the position in the operation screen. When the operation receiving unit 59 detects pressing of any of the plurality of hard keys provided in the hard key unit 167, the single execution unit 73 executes processing determined by the application program corresponding to the pressed hard key. .

  The single execution unit 73 includes an operation screen determination unit 79. The operation screen determination unit 79 determines an operation screen and outputs the operation screen to the display control unit 57 and the operation screen transmission unit 65. The operation screen determined by the operation screen determination unit 79 differs depending on the application program executed by the CPU 111. In the case of an IWS program, the operation screen is an operation screen described in a markup language. In the case of an API program or a native program, the operation of image data is performed. It is a screen.

  The display control unit 57 displays an operation screen on the display unit 161. The operation screen transmission unit 65 transmits the operation screen to the portable information device 200 that is a remote operation device. Thereby, the operation screen is displayed by the portable information device 200. When the operation screen is described in a markup language, the display control unit 57 generates an image of the operation screen according to the markup language and displays it on the display unit 161. When the operation screen is image data, the display control unit 57 The image is displayed on the display unit 161. When the operation screen is described in a markup language, the operation screen transmission unit 65 transmits the operation screen data described in the markup language to the portable information device 200 that is a remote operation device, and the operation screen is an image. In the case of data, the image data is transmitted to the portable information device 200 which is a remote control device.

  The identifying unit 55 identifies an application program to be executed by the process executing unit 53 and outputs program identification information of the identified application program to the used hard key determining unit 61.

  The used hard key determination unit 61 determines one or more hard keys determined by the application program executed by the process execution unit 53 among the plurality of hard keys provided in the operation unit 163 as the used hard key. The used hard key determining unit 61 outputs hard key identification information for identifying the determined used hard key to the hard key information transmitting unit 67. The used hard key determination unit 61 determines an application program specified by the program identification information input from the specifying unit 55 as an application program to be executed by the process execution unit 53. The use hard key determination unit 61 includes a language analysis unit 81, a program analysis unit 83, and a hard key information acquisition unit 85.

  If the type of application program specified by the program identification information input from the specifying unit 55 is a native program, the used hard key determining unit 61 excludes the power key 451 from among the plurality of hard keys provided in the hard key unit 167. All of one or more hard keys are determined to be used hard keys.

  The use hard key determination unit 61 includes a language analysis unit 81 and a program analysis unit 83 depending on whether the application program specified by the program identification information input from the specifying unit 55 is an API program, an IWS program, or a remote operation program. And the hard key information acquisition unit 85 are activated. The use hard key determination unit 61 activates the language analysis unit 81 if the type of the application program specified by the program identification information input from the specification unit 55 is an IWS program, and activates the program analysis unit 83 if the type is an API program. If it is a remote operation program, the hard key information acquisition unit 85 is activated.

  The language analysis unit 81 receives the operation screen described in the markup language from the operation screen determination unit 79, and determines the hard key to be used by analyzing the markup language description on the operation screen. Accordingly, the language analysis unit 81 determines a hard key to be used for each operation screen.

  The program analysis unit 83 receives the program identification information from the specifying unit 55, and determines the hard key to be used by analyzing the application program specified by the program identification information. The program analysis unit 83 includes an application unit determination unit 89 and a screen unit determination unit 87. The application unit determination unit 89 determines all hard keys used in the application program as used hard keys. The screen unit determination unit 87 receives an operation screen from the operation screen determination unit 79. The screen unit determination unit 87 determines, among the hard keys used in the application program, the hard key that is used while the operation screen input from the operation screen determination unit 79 is displayed as the used hard key. The application unit determining unit 89 determines all hard keys used in the application program as used hard keys, whereas the screen unit determining unit 87 is a hard key used while the operation screen is displayed. Use hard key to determine. In the application program to be met, hard key identification information of each hard key included in the used hard key may be described in advance for each operation screen.

  Hard key information acquisition unit 85 receives server identification information from server remote operation screen receiving unit 77. The hard key information acquisition unit 85 controls the communication I / F unit 112 to access the server using the server identification information, and acquires the used hard key information from the server. The used hard key information includes hard key identification information of a hard key defined by an application program executed by the server. The hard key information acquisition unit 85 outputs the hard key identification information of the acquired used hard key information to the hard key information transmission unit 67.

  The used hard key determination unit 61 activates the program analysis unit 83 instead of the language analysis unit 81 when the application program specified by the program identification information input from the specification unit 55 is an IWS program. May be. In this case, the language analysis unit 81 is not necessary. Further, the used hard key determining unit 61 may activate the program analyzing unit 83 when the application program specified by the program identification information input from the specifying unit 55 is a native program. In this case, hard key identification information of each hard key included in the used hard key may be described in advance in the native program for each operation screen.

  FIG. 7 is a block diagram illustrating an example of an outline of functions of a CPU included in the portable information device according to the first embodiment. The functions shown in FIG. 7 are functions formed in the CPU 201 when the CPU 201 included in the portable information device 200 executes a remote operation program stored in the flash memory 203 or the CD-ROM 211A. The remote operation program includes a screen generation program. Referring to FIG. 7, CPU 201 includes a remote operation unit 221 that remotely controls MFP 100 and a screen generation unit 241.

  When the user operates operation unit 207 and inputs an instruction to remotely operate MFP 100, remote operation unit 221 controls wireless LAN I / F 208, transmits a connection request to MFP 100, and communicates with MFP 100. Establish a route. The remote operation unit 221 includes an operation screen reception unit 223, a hard key information reception unit 225, a position detection unit 227, a position conversion unit 229, a key information conversion unit 231, a remote operation screen display unit 233, and an instruction transmission. Part 235.

  When a communication path is established with MFP 100, operation screen reception unit 223 controls wireless LAN I / F 208 to receive an operation screen from MFP 100, and the received operation screen is displayed as screen generation unit 241 and position conversion unit 229. Output to.

  When a communication path is established with MFP 100 that is the device to be controlled, hard key information receiving unit 225 receives hard key information from MFP 100 via wireless LAN I / F 208 and generates the received hard key information on a screen. Output to the unit 241 and the key information conversion unit 231.

  The screen generation unit 241 receives an operation screen from the operation screen reception unit 223 and receives hard key information from the hard key information reception unit 225. The screen generation unit 241 includes a hard key screen generation unit 243 and a remote operation screen generation unit 245. The hard key screen generation unit 243 generates a hard key screen based on the hard key information, and outputs the generated hard key screen to the remote operation screen generation unit 245. The hard key screen generation unit 243 includes a key arrangement unit 251. The remote operation screen generation unit 245 generates a remote operation screen including an operation screen and a hard key screen, and outputs the generated remote operation screen to the remote operation screen display unit 233. Remote operation screen generation unit 245 includes an area determination unit 253 and a screen arrangement unit 255.

  The area determination unit 253 determines a first area where the operation screen is arranged on the remote operation screen and a second area where the hard key screen is arranged. The area determination unit 253 determines the first area for arranging the operation screen on the remote operation screen, and determines the remaining area as the second area for arranging the hard key screen. The area determination unit 253 outputs the second area to the hard key screen generation unit 243 and outputs the first area and the second area to the screen arrangement unit 255.

  The region determining unit 253 determines the minimum size of the second region based on the hard key identification information included in the hard key information, and determines the first region so that the second region having the determined minimum size is secured. To do. Specifically, for each of one or more hard key identification information included in the hard key information, a soft key image corresponding to the hard key determined by the hard key identification information is specified, and one or more soft key images are arranged. Is determined as the minimum size of the second region. The minimum size of the second region can be represented by an area. When the size of one or more soft key images is the same, the minimum size of the second region can be determined based on the number of one or more soft key images. The minimum size of the second area is determined by using the size of the soft key image having the maximum size in the vertical and horizontal directions among one or more soft key images respectively corresponding to the one or more hard key identification information. You may make it define the minimum size of each of the vertical direction or horizontal direction of 2 area | regions. In this case, the minimum size of the second region includes the area and the minimum length in the vertical and horizontal directions.

  The area determination unit 253 determines the first area based on the minimum size of the second area and the direction of the display unit 206. Since the portable information device 200 is used while being supported by the user's hand, the direction of the display unit 206 with respect to the user's line of sight varies. For example, if the display unit 206 is rectangular, the display unit 206 with respect to the user's line of sight may be in a vertically long direction or in a horizontally long direction. The region determination unit 253 detects the attitude of the portable information device 200 with respect to the gravity direction based on, for example, the gravity direction detected by a gravity sensor or the like, and based on the attitude of the portable information device 200, the direction of the display unit 206 with respect to the user's line of sight To decide.

  Then, the area determination unit 253 determines the size of the display surface of the display unit 206 as the size of the remote operation screen, and determines the direction of the remote operation screen as the same direction as the direction of the display unit 206 with respect to the user's line of sight. Furthermore, the area determination unit 253 determines the first area so that the direction of the operation screen is the same as the direction of the remote operation screen. In other words, the size of the first area is determined such that the first area is similar to the operation screen, and the area excluding the first area from the remote operation screen is equal to or larger than the minimum size of the second area. The area determination unit 253 determines the first area and determines the area excluding the first area from the remote operation screen as the second area. The area determination unit 253 outputs the determined first area and second area to the screen arrangement unit 255 and outputs the second area to the key arrangement unit 251.

  The hard key screen generation unit 243 receives hard key information from the hard key information reception unit 225 and receives a second region from the region determination unit 253. The hard key screen generation unit 243 includes a key arrangement unit 251. The key arrangement unit 251 generates a hard key screen by arranging one or more soft key images respectively corresponding to one or more hard keys defined by the hard key information on the hard key screen having the same size as the second area. Then, the generated hard key screen is output to the screen arrangement unit 255 and the position conversion unit 229. When the operation screen is described in a markup language, the hard key screen generation unit 243 generates a hard key screen described in the markup language. In this case, the hard key screen includes a description for displaying a soft key image and a description for outputting key identification information in response to an instruction of the soft key image.

  The screen arrangement unit 255 receives the first region and the second region from the region determination unit 253, receives the operation screen from the operation screen reception unit 223, and receives the hard key screen from the hard key screen generation unit 243. The screen placement unit 255 creates a remote operation screen by placing an operation screen in the first area of the remote operation screen and placing a hard key screen in the second area, and the generated remote operation screen, first area The second area is output to the remote operation screen display unit 233. When the operation screen and the first area have different sizes, the screen layout unit 255 changes the size of the operation screen so that the size of the operation screen is the same as that of the first area. Are arranged in the first region. When the operation screen is described in the markup language, the screen layout unit 255 also describes the hard key screen in the markup language, so that the operation screen is arranged in the first area and the hard key screen is the second. A remote operation screen including a description of the markup language arranged in the area is generated.

  The remote operation screen display unit 233 displays the remote operation screen input from the remote operation screen generation unit 245 on the display unit 206, and outputs the remote operation screen, the first area, and the second area to the position conversion unit 229.

  The position detection unit 227 controls the touch panel 207B, and when the touch panel 207B detects the position designated by the user, the position detection unit 227 specifies the position on the display surface of the display unit 206 based on the detected position, and the specified position Is output to the position conversion unit 229.

  The position conversion unit 229 receives the remote operation screen, the first region and the second region from the remote operation screen display unit 233, receives the operation screen from the operation screen reception unit 223, and receives position information from the position detection unit 227. The hard key screen is input from the hard key screen generation unit 243. In the remote operation screen, the first area includes an operation screen, and the second area includes a hard key screen.

  The position conversion unit 229 specifies a position in the remote operation screen based on the position information. When the specified position is in the first area of the remote operation screen, the position conversion unit 229 specifies the position in the first area, and converts the specified position into a position in the operation screen. When the operation screen includes one or more components, the operation screen may be converted into the position of the component existing at the position specified in the operation screen. The position conversion unit 229 outputs position information indicating the position converted to the position in the operation screen to the instruction transmission unit 235. When the operation screen is described in a markup language, the command assigned to the position in the operation screen is output to the instruction transmission unit 235.

  When the position on the remote operation screen specified based on the position information is within the second area of the remote operation screen, the position conversion unit 229 displays a hard key screen and position information indicating the position in the second area. The data is output to the key information conversion unit 231.

  The key information conversion unit 231 receives hard key information from the hard key information reception unit 225, and receives a hard key screen and position information indicating a position in the second area from the position conversion unit 229. The key information conversion unit 231 specifies a soft key image arranged at a position specified by position information indicating a position in the second area from one or more soft key images included in the hard key screen. The key information conversion unit 231 specifies key identification information corresponding to the specified soft key image, and outputs the specified key identification information to the instruction transmission unit 235.

  The instruction transmission unit 235 includes a case where position information is input from the position conversion unit 229, a case where a command is input, and a case where key identification information is input from the key information conversion unit 231. When position information is input, instruction transmission unit 235 transmits a remote operation including the position information to MFP 100 via wireless LAN I / F 208. Further, when a command is input, the instruction transmission unit 235 transmits a remote operation including the command to the MFP 100 via the wireless LAN I / F 208. In this case, the remote operation is transmitted using the HTTP protocol. In addition, when key identification information is input, instruction transmission unit 235 transmits a remote operation including the key identification information to MFP 100 via wireless LAN I / F 208.

  FIG. 8 is a first diagram illustrating an example of the operation screen. Referring to FIG. 8, operation screen 500 shows an example of an operation screen when CPU 111 executes a native program. With respect to the operation screen 500, the hard keys that can be received while the operation screen 500 is displayed are all of the hard keys provided in the hard key unit 167 except for the power key 451.

  FIG. 9 is a first diagram illustrating an example of a remote operation screen. A remote operation screen 611 shown in FIG. 9 shows a remote operation screen when the portable information device 200 is in the landscape orientation. Referring to FIG. 9, remote operation screen 611 includes a first area 613 and a second area 615. The first area 613 includes an operation screen 500A, and the second area 615 includes a hard key screen 617. The operation screen 500A is an image obtained by scaling the operation screen 500 shown in FIG. 8 vertically and horizontally. Note that the operation screen 500A is preferably similar to the operation screen 500 shown in FIG. The hard key screen 617 is a screen in which a plurality of soft key images respectively corresponding to all of the hard keys provided in the hard key unit 167 except for the power key 451 are arranged. Specifically, a plurality of keys classified into the first to fourth groups 410, 420, 430, and 440 are arranged.

  FIG. 10 is a second diagram illustrating an example of the remote operation screen. A remote operation screen 621 shown in FIG. 10 shows a remote operation screen when the attitude of the portable information device 200 is in the vertically long direction. Referring to FIG. 10, remote operation screen 621 includes a first area 623 and a second area 625. The first area 623 includes an operation screen 500B, and the second area 625 includes a hard key screen 627. The operation screen 500B is an image obtained by scaling the operation screen 500 shown in FIG. 8 vertically and horizontally. Note that the operation screen 500B is preferably similar to the operation screen 500 shown in FIG. The hard key screen 627 is a screen on which a plurality of soft key images respectively corresponding to all of the hard keys provided in the hard key unit 167 except for the power key 451 are arranged. Specifically, a plurality of keys classified into the first to fourth groups 410, 420, 430, and 440 are arranged.

  FIG. 11 is a second diagram illustrating an example of the operation screen. Referring to FIG. 11, operation screen 510 shows an example of an operation screen when CPU 111 executes an API program. With respect to the operation screen 510, the hard key that can be received while the operation screen 510 is displayed is only the start key 431 among the plurality of hard keys provided in the hard key unit 167.

  FIG. 12 is a third diagram illustrating an example of a remote operation screen. A remote operation screen 651 shown in FIG. 12 is a remote operation screen when the portable information device 200 is in the landscape orientation. Referring to FIG. 12, remote operation screen 651 includes a first area 653 and a second area 655. The first area 653 includes an operation screen 510A, and the second area 655 includes a hard key screen 657. The operation screen 510A is similar to the operation screen 510 shown in FIG. The hard key screen 657 is a screen on which a soft key image corresponding to the start key 431 among a plurality of hard keys provided in the hard key unit 167 is arranged.

  FIG. 13 is a third diagram illustrating an example of the operation screen. Referring to FIG. 13, operation screen 520 shows an example of the operation screen when CPU 111 executes a remote operation program. In this case, operation screen 520 is generated by the server by causing server remotely controlled by MFP 100 to execute a server program including a description of an API command. The hard keys that can be received while the operation screen 520 is displayed with respect to the operation screen 520 are a numeric keypad 422 and a start key 431 among a plurality of hard keys provided in the hard key unit 167.

  FIG. 14 is a fourth diagram illustrating an example of a remote operation screen. A remote operation screen 661 shown in FIG. 14 shows the remote operation screen when the portable information device 200 is in the landscape orientation. Referring to FIG. 14, remote operation screen 661 includes a first area 663 and a second area 665. The first area 663 includes an operation screen 520A, and the second area 665 includes a hard key screen 667. The operation screen 520A is similar to the operation screen 520 shown in FIG. The hard key screen 667 is a screen in which soft key images respectively corresponding to the ten key 422 and the start key 431 among the plurality of hard keys provided in the hard key unit 167 are arranged.

  FIG. 15 is a fourth diagram illustrating an example of the operation screen. Referring to FIG. 15, operation screen 530 shows an example of the operation screen when CPU 111 executes the IWS program. In this case, the operation screen 530 is an image described in a markup language and generated by the CPU 111 executing a browsing program. A hard key that can be received while the operation screen 530 is displayed with respect to the operation screen 530 is the numeric keypad 422 among a plurality of hard keys provided in the hard key unit 167.

  FIG. 16 is a fifth diagram illustrating an example of a remote operation screen. A remote operation screen 671 shown in FIG. 16 shows a remote operation screen when the attitude of the portable information device 200 is in the horizontally long direction. Referring to FIG. 16, remote operation screen 671 includes a first area 673 and a second area 675. First region 673 includes an operation screen 530A, and second region 675 includes a hard key screen 677. The operation screen 530A is similar to the operation screen 530 shown in FIG. The hard key screen 677 is a screen on which soft key images corresponding to the numeric keypad 422 among a plurality of hard keys included in the hard key unit 167 are arranged.

  FIG. 17 is a flowchart illustrating an example of the flow of remote operation processing according to the first embodiment. The remote operation process is a process executed by the CPU 201 when the CPU 201 included in the portable information device 200 executes a remote operation program stored in the flash memory 203 or the CD-ROM 211A. The remote operation program includes a screen generation program. Referring to FIG. 17, CPU 201 determines whether a remote operation instruction has been accepted (step S01). If the operation unit 207 receives a remote operation instruction input by the user, the remote operation instruction is received. The process waits until a remote operation instruction is accepted (NO in step S01). If a remote operation instruction is accepted (YES in step S01), the process proceeds to step S02.

  In step S02, a connection request is transmitted to MFP 100. The wireless LAN I / F 208 is controlled to transmit a connection request in order to establish a communication path with the MFP 100. Then, it is determined whether or not a communication path has been established (step S03). A communication path is established by negotiating with the MFP 100. Step S02 is repeated until the communication path is established (NO in step S03). If the communication path is established (YES in step S03), the process proceeds to step S04.

  In step S04, the control target screen size is acquired from MFP 100. The control target screen size is acquired from MFP 100 via the communication path established in step S03. The control target screen size is the size of the display surface of the display unit 161 provided in the MFP 100. The control target screen size is the same as the size of the operation screen displayed on the display unit 161 by the MFP 100.

  In the next step S05, the posture is detected. Based on the triaxial acceleration output from the acceleration sensor 210, the attitude of the portable information device 200 is detected. The orientation of the portable information device 200 is such that the longitudinal direction of the display unit 206 is closer to the gravitational direction than the short side direction, and the lateral direction is such that the short side direction of the display unit 206 is closer to the gravitational direction than the longitudinal direction. One of them. In step S06, the display screen size is acquired. The display screen size is the size of the display surface of the display unit 206. In the next step S07, the direction of the display screen is determined based on the posture detected in step S05. If the orientation of the portable information device 200 is in the vertical direction, the direction of the display screen is determined to be the portrait orientation in which the display screen is vertically long. If the orientation of the portable information device 200 is in the horizontal direction, the direction of the display screen is displayed. The screen is determined in the horizontal direction where the screen is horizontally long.

  In step S08, it is determined whether MFP 100 has disconnected the communication path established in step S03. If the communication path is disconnected, the process ends. If not, the process proceeds to step S09.

  In step S09, it is determined whether an operation screen has been received. The wireless LAN I / F 208 is controlled to receive an operation screen from the MFP 100. If the operation screen is received, the process proceeds to step S10; otherwise, the process returns to step S08. In step S10, hard key information is received from portable information device 200, and the process proceeds to step S11. The wireless LAN I / F 208 is controlled to receive hard key information from the MFP 100. The hard key information includes one or more hard key identification information for identifying one or more hard keys that can be accepted by MFP 100 while the operation screen received in step S09 is displayed on MFP 100.

  In the next step S11, a screen generation process is executed, and the process proceeds to step S12. Although details of the screen generation process will be described later, this is a process of generating a remote operation screen to be displayed on the display unit 206 based on the operation screen and hard key information.

  In step S12, a remote operation screen is displayed on display unit 206, and the process proceeds to step S13. In step S13, it is determined whether an instruction has been accepted. When the position indicated by the user is detected by touch panel 207B, the instruction is accepted. The process waits until an instruction is accepted (NO in step S13). If the instruction is accepted (YES in step S13), the process proceeds to step S14.

  In step S14, it is determined whether the detected position in the remote operation screen is within the first area. If so, the process proceeds to step S15; otherwise, the process proceeds to step S16. In step S15, the position on the remote operation screen is converted to the position on the operation screen, and the process proceeds to step S16. In step S16, it is determined whether the detected position in the remote operation screen is within the second area. If so, the process proceeds to step S17; otherwise, the process proceeds to step S18. In step S17, a hard key corresponding to the detected position is determined, and the process proceeds to step S18. Since a hard key screen is set in the second area of the remote operation screen, a soft key image including a position detected from a plurality of soft key images arranged on the hard key screen is specified and specified. Determine the hard key corresponding to the soft key image.

  In step S18, it is determined whether or not the received instruction is an end operation. If it is not an end operation, the process proceeds to step S19. If it is an end operation, the remote operation process is ended. In step S19, wireless LAN I / F 208 is controlled, a remote operation is transmitted to MFP 100, and the process returns to step S09. The remote operation includes position information indicating a position on the operation screen when step S15 is executed, in other words, when the user instructs the first area of the remote operation screen. The remote operation includes the key identification information of the hard key determined in step S17 when step S17 is executed, in other words, when the user designates the second area of the remote operation screen.

  FIG. 18 is a flowchart illustrating an example of the flow of the screen generation process. The screen generation process shown in FIG. 18 is a process executed in step S11 of FIG. Referring to FIG. 18, the minimum size of the second area is determined based on the hard key information. Specifically, for each of one or more hard key identification information included in the hard key information acquired in step S10 of the remote operation processing shown in FIG. 17, a soft key corresponding to the hard key determined by the hard key identification information An image is specified, and the minimum size in which the specified one or more soft key images can be arranged is determined as the minimum size of the second region. The minimum size of the second region can be represented by an area. When the size of one or more soft key images is the same, the minimum size of the second region can be determined based on the number of one or more soft key images. In addition, the minimum size of the second area is determined by using the size of the soft key image having the maximum size in the vertical direction and the horizontal direction of the one or more soft key images respectively corresponding to the one or more hard key identification information. The minimum length in the vertical or horizontal direction may be determined. In this case, the minimum size of the second region includes the area and the minimum length in the vertical and horizontal directions.

  In the next step S22, the first region is determined. The display screen size acquired in step S06 of the remote operation processing shown in FIG. 17, the direction of the display screen determined in step S07, the minimum size of the second area determined in step S21, and step S09. 1st area | region is determined based on the shape of the operation screen received in (1).

  The shape of the first region is similar to the shape of the operation screen. The size and direction of the remote operation screen are matched with the size and direction of the display screen, and a temporary area similar to the shape of the operation screen is arranged in the remote operation screen. At this time, the first area is changed by scaling the temporary area so that the area of the area excluding the temporary area from the remote operation screen, the length in the vertical direction and the lateral rear is equal to or larger than the minimum size of the second area. decide.

  In the next step S23, a region other than the first region determined in step S22 is determined as the second region from the remote operation screen for determining the second region. Then, a soft key image is acquired (step S24). For each of one or more hard key identification information included in the hard key information acquired in step S10 of the remote operation processing shown in FIG. 17, a soft key image corresponding to the hard key determined by the hard key identification information is acquired. Then, a hard key screen is generated. A hard key screen is generated by arranging one or more soft key images acquired in step S24 in an area having the same shape and size as the second area determined in step S23.

  In the next step S26, the size of the operation screen is changed. The size of the operation screen is changed so that the size of the operation screen is the same as the size of the first area determined in step S22.

  In the next step S27, a remote operation screen is generated, and the process returns to the remote operation process. The operation screen after the size change in step S26 is arranged in the first area, and the remote operation screen is generated by arranging the hard key screen generated in step S25 in the second area.

  FIG. 19 is a flowchart illustrating an example of the flow of remote control processing according to the first embodiment. The remote control process is a process executed by CPU 111 when CPU 111 provided in MFP 100 executes a remote control program stored in ROM 113, HDD 115, and CD-ROM 118.

  Referring to FIG. 19, CPU 111 determines whether a connection request has been received (step S101). It is determined whether communication I / F unit 112 has received a connection request from the outside. The process waits until a connection request is received. If a connection request is received, the process proceeds to step S102. Here, a case where a connection request is received from portable information device 200 will be described as an example.

  In step S102, a communication path is established with portable information device 200. Then, the operation screen is transmitted to portable information device 200 that is a remote operation device via communication I / F unit 112 (step S103), and the process proceeds to step S104.

  In step S104, a used hard key is determined. Among the plurality of hard keys included in the hard key unit 167, a hard key that can be accepted while the operation screen transmitted in step S103 is displayed on the display unit 161 is determined as a used hard key. Then, the hard key information is transmitted to the portable information device 200 (step S105). The hard key information includes hard key identification information for identifying the used hard key determined in step S104.

  In step S106, it is determined whether a remote operation has been received. The communication I / F unit 112 is in a standby state until it receives a remote operation from the portable information device 200 that is a remote operation device (NO in step S106). If the remote operation is received (YES in step S106), the process is stepped. Proceed to S107. In step S107, it is determined whether the received remote operation includes position information. If the remote operation includes position information, the process proceeds to step S108; otherwise, the process proceeds to step S109. In step S108, the remote operation is specified from the position specified by the position information included in the remote operation in the operation screen transmitted to portable information device 200 in step S103 or step S114 described later, and the process proceeds to step S110. Proceed. For example, if the position specified in the operation screen is within the transition button, the screen transition operation is specified. If the position specified in the operation screen is within the area to which the command for setting the setting value is assigned, the setting operation is specified. If the position specified on the operation screen is within the area to which the execution command is assigned, the execution instruction operation is specified.

  If the process proceeds to step S109, the received remote instruction includes key identification information. In step S109, the remote operation associated with the hard key specified by the key identification information included in the remote instruction is specified, and the process proceeds to step S110.

  In step S110, the process is branched by the specified remote operation. If the remote operation is a screen transition operation for switching the display screen, the process proceeds to step S111. If the remote operation is a setting operation for setting a setting value, the process proceeds to step S115, and the remote operation instructs the execution of the process. If it is an operation, the process proceeds to step S116, and if the remote operation is an end operation to end the remote control, the process proceeds to step S117.

  In step S111, an operation screen is specified, and the process proceeds to step S112. In step S112, the identified operation screen is transmitted to portable information device 200, which is a remote operation device, via communication I / F unit 112. Then, the used hard key is determined (step S113). Among the plurality of hard keys included in the hard key unit 167, the hard key that can be accepted while the operation screen transmitted in step S112 is displayed on the display unit 161 is determined as the used hard key. Then, the hard key information is transmitted to portable information device 200 (step S114), and the process returns to step S106. The hard key information includes hard key identification information for identifying the used hard key determined in step S104.

  In step S115, the setting value specified by the setting operation is changed, and the process returns to step S106. In step S116, the process specified by the execution instruction operation is executed according to the set value set at that time, and the process returns to step S106. In step S117, the communication path established in step S102 is disconnected, and the process ends.

  As described above, in remote control system 1 according to the first embodiment, MFP 100 causes display unit 161 to display an application program executed by CPU 111 of MFP 100 from among a plurality of application programs that can be executed by MFP 100. The application program corresponding to the screen is specified, and one or more hard keys determined by the specified application program among the plurality of hard keys provided in the operation unit 163 are determined as the use hard keys. Portable information device 200 generates a hard key screen on which soft key images corresponding to one or more used hard keys determined by MFP 100 are arranged, generates a remote operation screen including the operation screen and the hard key screen, and displays it. To do. Therefore, a remote operation screen capable of designating a hard key defined by an application program corresponding to the operation screen displayed on MFP 100 among the plurality of hard keys provided in MFP 100 can be displayed on portable information device 200. .

  MFP 100 includes one or more hard keys excluding the power key among the plurality of hard keys provided in operation unit 163, and one or more hard keys that can be received by MFP 100 while CPU 111 executes the application program. Use hard key to determine. Since the soft key images included in the hard key screen correspond to the hard keys defined by the application program executed on MFP 100, the number of soft key images can be reduced. Therefore, the portable information device 200 can display a remote operation screen in which the size of the operation screen is increased by reducing the size of the hard key screen as much as possible.

  In addition, MFP 100 uses one or more hard keys that can be received while the operation screen is displayed among the one or more hard keys excluding the power key among the plurality of hard keys provided in operation unit 163 as the use hard keys. decide. For this reason, the size of the hard key screen can be made as small as possible.

  Further, MFP 100 executes an application program, displays an operation screen, transmits the operation screen to portable information device 200, and is identified by a remote operation received from portable information device 200 among a plurality of application programs. Run the program. Therefore, the application program can be executed by remotely operating MFP 100 from portable information device 200, and the same operation screen as the operation screen displayed on MFP 100 can be displayed on portable information device 200.

  Further, when displaying an operation screen described in a markup language, MFP 100 analyzes the operation screen to determine a hard key that can be accepted while the operation screen is displayed as a used hard key. For this reason, the used hard key can be determined only from the operation screen.

  Further, when the MFP 100 displays an operation screen received from an external server and remotely operates the server by transmitting a remote operation to the server, the MFP 100 is displayed by specifying an application program to be executed by the external server. Identify the application program that corresponds to the current operation screen. Therefore, when remotely controlling the server from MFP 100, it is possible to remotely operate MFP 100 and the server from portable information device 200.

  Further, when MFP 100 remotely operates the server, MFP 100 obtains information on the hard key from an external server, so that a hard key screen including hard keys necessary for remotely operating the server can be displayed. .

<Second Embodiment>
The remote control system 1 according to the first embodiment generates a remote operation screen on the portable information device 200 based on the operation screen displayed on the MFP 100. In remote control system 1 in the second embodiment, MFP 100 generates a remote operation screen based on an operation screen. Hereinafter, the remote control system 1A in the second embodiment will be described.

  The overall outline of the remote control system in the second embodiment is the same as the overall outline of the remote control system in the first embodiment shown in FIG. Also, the overview and hardware configuration of MFP 100 in the second embodiment are the same as the overview shown in FIG. 2 and the hardware configuration shown in FIG. Furthermore, the hardware configuration of portable information devices 200, 200A, and 200B in the second embodiment is the same as the hardware configuration shown in FIG. Therefore, these descriptions will not be repeated.

  FIG. 20 is a block diagram illustrating an example of functions of the CPU provided in the MFP according to the second embodiment. The functions shown in FIG. 20 are functions formed in CPU 111 when CPU 111 provided in MFP 100 according to the second embodiment executes a remote control program stored in ROM 113, HDD 115, and CD-ROM 118. The remote control program includes a screen generation program. Referring to FIG. 19, the difference from the function shown in FIG. 6 is that remote control unit 51 is changed to remote control unit 51A, and screen generation unit 91 is added. The other functions are the same as those shown in FIG. 6, and thus description thereof will not be repeated here.

  The remote control unit 51A executes processing according to the remote operation received from the remote operation device. The remote control unit 51 controls the communication I / F unit 112, receives a connection request from the remote operation device, and establishes a communication path with the remote operation device. Here, a case where a connection request is received from portable information device 200 that is a remote control device will be described as an example. In this case, the remote control unit 51A establishes a communication path with the portable information device 200, transmits a remote operation screen to the portable information device 200 by a request from the portable information device 200, and receives it from the portable information device 200. The process is executed according to the remote operation.

  The screen generation unit 91 receives hard key information from the used hard key determination unit 61 and receives an operation screen from the process execution unit 53. When the CPU 111 executes an application program for controlling the MFP 100, the single execution unit 73 is activated, so that the screen generation unit 91 receives hard key information from the language analysis unit 81 or the program analysis unit 83, and operates An operation screen is input from the screen determination unit 79. Further, when the CPU 111 executes an application program for remotely operating a server to be remotely operated, the server remote operation unit 71 is activated, so that the screen generation unit 91 receives the hard key information from the hard key information acquisition unit 85. , And the server remote operation screen receiving unit 77 inputs the server remote operation screen. The screen generation unit 91 processes the server remote operation screen as an operation screen.

  Screen generation unit 91 includes a hard key screen generation unit 93 and a remote operation screen generation unit 95. The hard key screen generation unit 93 generates a hard key screen based on the hard key information, and outputs the generated hard key screen to the remote operation screen generation unit 95. The hard key screen generation unit 93 includes a key arrangement unit 97. Remote operation screen generation unit 95 generates a remote operation screen including an operation screen and a hard key screen, and outputs the generated remote operation screen to remote operation screen transmission unit 65A. Remote operation screen generation unit 95 includes an area determination unit 98 and a screen layout unit 99.

  The area determination unit 98 determines a first area where the operation screen is arranged on the remote operation screen and a second area where the hard key screen is arranged. The area determining unit 98 determines a first area for arranging the operation screen on the remote operation screen, and determines the remaining area as a second area for arranging the hard key screen. The area determination unit 98 outputs the second area to the hard key screen generation unit 93, and outputs the first area and the second area to the screen arrangement unit 99.

  The area determining unit 98 determines the minimum size of the second area based on the hard key identification information included in the hard key information, and determines the first area so that the second area having the determined minimum size is secured. To do. Specifically, for each of one or more hard key identification information included in the hard key information, a soft key image corresponding to the hard key determined by the hard key identification information is specified, and one or more soft key images are arranged. Is determined as the minimum size of the second region. The minimum size of the second region can be represented by an area. When the size of one or more soft key images is the same, the minimum size of the second region can be determined based on the number of one or more soft key images. The minimum size of the second area is determined by using the size of the soft key image having the maximum size in the vertical and horizontal directions among one or more soft key images respectively corresponding to the one or more hard key identification information. You may make it define the minimum size of each of the vertical direction or horizontal direction of 2 area | regions. In this case, the minimum size of the second region includes the area and the minimum length in the vertical and horizontal directions.

  The size of the display unit varies depending on the remote operation device on which the remote operation screen is displayed. Since the remote operation device is supported and used by the user's hand, the direction of the display unit with respect to the user's line of sight varies. Therefore, the area determination unit 253 acquires the size of the display unit 206 of the portable information device 200 and the attitude of the portable information device 200 from the portable information device 200 that is a remote operation device, and determines the size and direction of the remote operation screen. . Then, the area determination unit 98 determines the first area so that the direction of the operation screen is the same as the direction of the remote operation screen. Specifically, the size of the first area is determined so that the first area is similar to the operation screen, and the area excluding the first area from the remote operation screen is equal to or larger than the minimum size of the second area. The area determining unit 98 determines the first area and determines the area excluding the first area from the remote operation screen as the second area. The area determination unit 98 outputs the determined first area and second area to the screen arrangement unit 99 and outputs the second area to the key arrangement unit 251.

  The hard key screen generation unit 93 receives hard key information from the used hard key determination unit 61 and receives a second region from the region determination unit 98. The hard key screen generation unit 93 includes a key arrangement unit 97. The key arrangement unit 97 generates a hard key screen by arranging one or more soft key images respectively corresponding to one or more hard keys defined by the hard key information on the hard key screen having the same size as the second area. Then, the generated hard key screen is output to the screen layout unit 99. When the operation screen is described in a markup language, the hard key screen generation unit 93 generates a hard key screen described in the markup language. In this case, the hard key screen includes a description for displaying a soft key image and a description for outputting key identification information in response to an instruction of the soft key image.

  The screen arrangement unit 99 receives the first region and the second region from the region determination unit 98, receives the operation screen from the server remote operation unit 71 or the single execution unit 73, and receives the hard key screen from the hard key screen generation unit 93. Entered. The screen layout unit 99 generates a remote operation screen by arranging an operation screen in the first area of the remote operation screen and arranging a hard key screen in the second area. The generated remote operation screen, first area The second area is output to the remote control unit 51A. When the size of the operation screen is different from that of the first area, the screen layout unit 99 changes the size of the operation screen so that the size of the operation screen becomes the same as that of the first area. Are arranged in the first region. When the operation screen is described in a markup language, the screen layout unit 99 also describes the hard key screen in the markup language, so that the operation screen is disposed in the first area and the hard key screen is the second. A remote operation screen including a description of the markup language arranged in the area is generated.

  The remote control unit 51A includes a remote operation reception unit 63A and a remote operation screen transmission unit 65A. The remote operation screen transmission unit 65A transmits the remote operation screen input from the screen generation unit 91 to the portable information device 200 via the communication I / F unit 112. Thereby, the remote operation screen is displayed on the portable information device 200.

  FIG. 21 is a block diagram illustrating an example of detailed functions of the remote operation receiving unit 63A. Referring to FIG. 21, remote operation receiving unit 63 </ b> A includes an instruction receiving unit 21, a position converting unit 23, a key information converting unit 25, and an instruction determining unit 27. The instruction receiving unit 21 controls the communication I / F unit 112, receives a remote operation from the portable information device 200, and outputs the received remote operation to the position conversion unit 23. The remote operation includes position information indicating the position in the remote operation screen transmitted by the remote operation screen transmission unit 65A to the portable information device 200.

  The position converting unit 23 specifies the position in the remote operation screen in response to the position information input from the instruction receiving unit 21. When the specified position is within the first region of the remote operation screen, the position conversion unit 23 specifies the position in the operation screen and converts the specified position into a position in the operation screen. Conversion to a position in the same component as the component existing at the position specified in the operation screen is performed. The position conversion unit 229 outputs position information indicating the position converted to the position in the operation screen to the instruction determination unit 27. When the operation screen is described in a markup language, the command assigned to the position in the operation screen is output to the instruction determination unit 27.

  If the position in the remote operation screen specified based on the position information is within the second area of the remote operation screen, the position conversion section 23 outputs position information indicating the position in the second area to the key information conversion section 231. To do.

  The key information conversion unit 25 receives the position information indicating the position in the second area from the position conversion unit 23, and selects one of the one or more soft key images included in the hard key screen from the second area. The soft key image arranged at the position specified by the position information indicating the position of is specified. The key information conversion unit 25 specifies key identification information corresponding to the specified soft key image, and outputs the specified key identification information to the instruction determination unit 27. When the operation screen is described in a markup language, a command including the specified key identification information is output to the instruction determination unit 27.

  The instruction determination unit 27 includes a case where position information is input from the position conversion unit 23, a case where a command is input, and a case where key identification information is input from the key information conversion unit 231. When the position information is input, the instruction determination unit 27 outputs a remote operation including the position information to the process execution unit 53. In addition, when a command is input, the instruction determination unit 27 outputs a remote operation including the command to the process execution unit 53. In addition, when key identification information is input, the instruction determination unit 27 outputs a remote operation including the key identification information to the process execution unit 53.

  FIG. 22 is a block diagram illustrating an example of an outline of functions of a CPU provided in the portable information device according to the second embodiment. The functions shown in FIG. 22 are functions formed in the CPU 201A when the CPU 201A included in the portable information device 200 according to the second embodiment executes a remote operation program stored in the flash memory 203 or the CD-ROM 211A. It is. Referring to FIG. 22, CPU 201A includes a remote operation screen reception unit 291, a remote operation screen display unit 287, a position detection unit 281, and an instruction transmission unit 289.

  When remote control screen receiving unit 291 establishes a communication path with MFP 100, remote control screen receiving unit 291 controls wireless LAN I / F 208 to receive a remote control screen from MFP 100, and displays the received remote control screen as remote control screen display unit 287. And output to the position detector 281. The remote operation screen display unit 287 displays the remote operation screen input from the remote operation screen reception unit 291 on the display unit 206.

  The position detection unit 281 controls the touch panel 207B, and when the touch panel 207B detects the position designated by the user, the position in the remote operation screen input from the remote operation screen reception unit 291 based on the detected position. Is identified. The position detection unit 281 outputs position information indicating the position on the remote operation screen to the instruction transmission unit 289. Instruction transmission unit 289 transmits a remote instruction including the position information input from position detection unit 281 to MFP 100 via wireless LAN I / F 208.

  FIG. 23 is a flowchart illustrating an example of a flow of remote operation processing according to the second embodiment. The difference from the remote operation processing in the first embodiment shown in FIG. 17 is that step S05A and step S06A are added, step S09 is changed to step S09A, step S04, step S07, and step S10. Step S11 and Steps S14 to S17 are deleted. The other processes are the same as the remote operation process in the first embodiment shown in FIG. Hereinafter, differences from the remote operation process in the first embodiment shown in FIG. 17 will be mainly described.

  In step S05A, the attitude detected in step S05 is transmitted to MFP 100 via wireless LAN I / F 208, and the process proceeds to step S06. In step S06A, the display screen size acquired in step S06 is transmitted to MFP 100 via wireless LAN I / F 208, and the process proceeds to step S08.

  In step S08, it is determined whether MFP 100 has disconnected the communication path established in step S03. If the communication path is disconnected, the process ends. If not, the process proceeds to step S09A.

  In step S09A, it is determined whether a remote operation screen has been received. The wireless LAN I / F 208 is controlled to receive a remote operation screen from the MFP 100. If the remote operation screen is received, the process proceeds to step S12. If not, the process returns to step S08.

  In step S12, a remote operation screen is displayed on display unit 206, and the process proceeds to step S13. In step S13, it is determined whether an instruction has been accepted. The process waits until an instruction is accepted (NO in step S13). If the instruction is accepted (YES in step S13), the process proceeds to step S18. In step S18, it is determined whether or not the received instruction is an end operation. If it is not an end operation, the process proceeds to step S19. If it is an end operation, the remote operation process is ended. In step S19, a remote operation including position information indicating the position specified by the instruction received in step S13 is transmitted to MFP 100 by controlling wireless LAN I / F 208, and the process returns to step S09.

  FIG. 24 is a flowchart illustrating an example of a flow of remote control processing according to the second embodiment. The remote control process is a process executed by the CPU 111 when the CPU 111 provided in the MFP 100 according to the second embodiment executes a remote control program stored in the ROM 113, the HDD 115, and the CD-ROM 118. The remote control program includes a screen generation program. The same processes as those shown in FIG. 19 are denoted by the same reference numerals.

  Referring to FIG. 24, CPU 111 determines whether a connection request has been received (step S101). It is determined whether communication I / F unit 112 has received a connection request from the outside. The process waits until a connection request is received. If a connection request is received, the process proceeds to step S102. Here, a case where a connection request is received from portable information device 200 will be described as an example.

  In step S102, a communication path is established with portable information device 200. Then, the control target screen size is acquired (step S121). The control target screen size is the size of the display surface of the display unit 161.

  In the next step S122, the display screen size is received from the portable information device 200 via the communication path established in step S102. The display screen size is the size of the display surface of the display unit 206 of the portable information device 200. In step S123, the attitude is received from portable information device 200 via the communication path established in step S102.

  In the next step S124, an operation screen is determined. In step S125, the hard key to be used is determined. The used hard key includes hard key identification information of each of one or more hard keys used while the operation screen determined in step S124 is displayed. The hard key information includes hard key identification information for each of one or more hard keys used by an application program executed by the CPU 111.

  In the next step S126, the screen generation process shown in FIG. 18 is executed, and the process proceeds to step S103A.

  In the next step S103A, the remote operation screen generated by executing the screen generation process is transmitted to the portable information device 200 via the communication path established in step S102. Thereby, the remote operation screen is displayed on the portable information device 200.

  In the next step S106, it is determined whether a remote instruction has been received. The communication I / F unit 112 is in a standby state until it receives a remote instruction from the portable information device 200, which is a remote control device (NO in step S106). If the remote instruction is received (YES in step S106), the process is stepped. Proceed to S127. In step S127, it is determined whether or not the position in the remote operation screen specified by the position information included in the received remote instruction is within the first area. If it is within the first region, the process proceeds to step S128; otherwise, the process proceeds to step S130.

  In step S128, the position on the remote operation screen is converted to the position on the operation screen, and the process proceeds to step S129. In step S129, the remote operation is specified from the position in the operation screen, and the process proceeds to step S130. For example, if the position specified in the operation screen is within the transition button, the screen transition operation is specified. If the position specified in the operation screen is within the area to which the command for setting the setting value is assigned, the setting operation is specified. If the position specified on the operation screen is within the area to which the execution command is assigned, the execution instruction operation is specified.

  In step S130, it is determined whether or not the position in the remote operation screen specified by the position information included in the received remote instruction is within the second area. If so, the process proceeds to step S131; otherwise, the process proceeds to step S110. In step S131, a hard key corresponding to the detected position is determined, and the process proceeds to step S132. Since a hard key screen is set in the second area of the remote operation screen, a soft key image including a position detected from a plurality of soft key images arranged on the hard key screen is specified and specified. Determine the hard key corresponding to the soft key image. In step S132, the remote operation associated with the determined hard key is specified, and the process proceeds to step S110.

  In step S110, the process is branched by the remote operation specified in step S129 or step S132. If the remote operation is a screen transition operation for switching the operation screen, the process proceeds to step S111. If the remote operation is a setting operation for setting a setting value, the process proceeds to step S115. The remote operation instructs the execution of the process. If so, the process proceeds to step S116. If the remote operation is an end operation to end the remote control, the process proceeds to step S117.

  In step S111, an operation screen is specified. Then, the used hard key is determined (step S113), and the process returns to step S126. The used hard key includes hard key identification information of each of one or more hard keys used while the operation screen specified in step S111 is displayed.

  In step S115, the setting value specified by the setting operation is changed, and the process returns to step S127. In step S116, the process specified by the execution instruction operation is executed according to the set value set at that time, and the process returns to step S126. In step S117, the communication path established in step S102 is disconnected, and the process ends.

  As described above, in remote control system 1 according to the second embodiment, MFP 100 displays an application program executed by CPU 111 of MFP 100 on display unit 161 from among a plurality of application programs that can be executed by MFP 100. The application program corresponding to the screen is specified, and one or more hard keys determined by the specified application program among the plurality of hard keys provided in the operation unit 163 are determined as the use hard keys. Further, MFP 100 generates a hard key screen on which soft key images corresponding to the determined one or more used hard keys are arranged, generates a remote operation screen including the operation screen and the hard key screen, and stores it in portable information device 200. Send. For this reason, it is possible to cause portable information device 200 to display a remote operation screen in which a hard key determined by an application program corresponding to the operation screen displayed on MFP 100 among the plurality of hard keys provided in MFP 100 can be specified. .

  MFP 100 includes one or more hard keys excluding the power key among the plurality of hard keys provided in operation unit 163, and one or more hard keys that can be received by MFP 100 while CPU 111 executes the application program. Use hard key to determine. Since the soft key images included in the hard key screen correspond to the hard keys defined by the application program executed on MFP 100, the number of soft key images can be reduced. Therefore, the portable information device 200 can display a remote operation screen in which the size of the operation screen is increased by reducing the size of the hard key screen as much as possible.

  In addition, MFP 100 uses one or more hard keys that can be received while the operation screen is displayed among the one or more hard keys excluding the power key among the plurality of hard keys provided in operation unit 163 as the use hard keys. decide. For this reason, the size of the hard key screen can be made as small as possible.

  Further, MFP 100 executes an application program, displays an operation screen, transmits the operation screen to portable information device 200, and is identified by a remote operation received from portable information device 200 among a plurality of application programs. Run the program. Therefore, the application program can be executed by remotely operating MFP 100 from portable information device 200, and the same operation screen as the operation screen displayed on MFP 100 can be displayed on portable information device 200.

  Further, when displaying an operation screen described in a markup language, MFP 100 analyzes the operation screen to determine a hard key that can be accepted while the operation screen is displayed as a used hard key. For this reason, the used hard key can be determined only from the operation screen.

  Further, when the MFP 100 displays an operation screen received from an external server and remotely operates the server by transmitting a remote operation to the server, the MFP 100 is displayed by specifying an application program to be executed by the external server. Identify the application program that corresponds to the current operation screen. Therefore, when remotely controlling the server from MFP 100, it is possible to remotely operate MFP 100 and the server from portable information device 200.

  Further, when MFP 100 remotely operates the server, MFP 100 obtains information on the hard key from an external server, so that a hard key screen including hard keys necessary for remotely operating the server can be displayed. .

  In the first embodiment described above, the MFP 100 has been described as an example of the image processing apparatus and the portable information apparatus 200 is described as an example of the remote operation apparatus. However, the remote operation processing illustrated in FIGS. As a remote control method that causes the CPU 201 that controls the portable information device 200 to execute the remote control process illustrated in FIG. 19 and that causes the CPU 111 that controls the MFP 100 to execute the remote control process. Needless to say, it can be captured. Similarly, in the second embodiment, the MFP 100 has been described as an example of an image processing apparatus and the portable information apparatus 200 is described as an example of a remote operation apparatus. However, the remote operation processing illustrated in FIG. 24 can be executed as a remote control method for causing the CPU 111 for controlling the MFP 100 to execute the remote control processing shown in FIG. 24, and a remote control program for causing the CPU 111 for controlling the MFP 100 to execute the remote control method. Needless to say.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1, 1A Remote control system, 3 networks, 5 wireless stations, 100 MFP, 200, 200A, 200B, portable information device, 110 main circuit, 111 CPU, 112 communication I / F unit, 113 ROM, 114 RAM, 115 HDD, 116 Facsimile unit, 117 external storage device, 120 automatic document feeder, 130 document reading unit, 140 image forming unit, 150 paper feeding unit, 155 post-processing unit, 160 operation panel, 161 display unit, 163 operation unit, 165 touch panel, 165 Hard key section, 167 Hard key section, 201 CPU, 202 Camera, 203 Flash memory, 204 Wireless communication section, 205 Call section, 206 Display section, 207 Operation section, 207A Main key, 207B Touch panel, 208 Wireless LAN I / F, 210 Accelerometer, 21 External storage device, 21 instruction receiving unit, 23 position converting unit, 25 key information converting unit, 27 instruction determining unit, 51, 51A remote control unit, 53 processing execution unit, 55 specifying unit, 57 display control unit, 59 operation receiving unit , 61 Used hard key determination unit, 63, 63A Remote operation reception unit, 65 Operation screen transmission unit, 65A Remote operation screen transmission unit, 67 Hard key information transmission unit, 71 Server remote operation unit, 73 Single execution unit, 75 Server remote Operation transmission unit, 77 Server remote operation screen reception unit, 79 Operation screen determination unit, 81 Language analysis unit, 83 Program analysis unit, 85 Hard key information acquisition unit, 87 Screen unit determination unit, 89 Application unit determination unit, 91 Screen generation Section, 93 hard key screen generation section, 95 remote operation screen generation section, 97 key arrangement section, 98 area determination section, 99 screen arrangement section, 221 remote operation 223 operation screen receiving unit, 225 hard key information receiving unit, 227 position detection unit, 229 position conversion unit, 231 key information conversion unit, 233 remote operation screen display unit, 235 instruction transmission unit, 241 screen generation unit, 243 hardware Key screen generation unit, 245 Remote operation screen generation unit, 251 Key arrangement unit, 253 Region determination unit, 255 Screen arrangement unit, 281 Position detection unit, 287 Remote operation screen display unit, 289 Instruction transmission unit, 291 Remote operation screen reception unit .

Claims (18)

A remote control system including an image processing device and a remote operation device for remotely operating the image processing device,
The image processing apparatus includes an operation receiving unit having a plurality of hard keys;
Display means for displaying an operation screen;
Identifying means for identifying an application program corresponding to an operation screen displayed on the display means from among a plurality of application programs;
Use hard key determining means for determining one or more hard keys defined by the application program specified by the specifying means among the plurality of hard keys as the use hard key;
One of the image processing device and the remote control device displays one or more soft key images corresponding to each of the one or more used hard keys based on the determined hard key information related to the one or more used hard keys. Hard key screen generating means for generating the arranged hard key screen;
A remote control system comprising: a remote operation screen generating means for generating a remote operation screen including the operation screen and the generated hard key screen.   The used hard key determining means is one or more that can be received by the operation receiving means while the specified application program is being executed among one or more hard keys excluding the power key among the plurality of hard keys. The remote control system according to claim 1, further comprising an application unit determining unit that determines a hard key to be used as a used hard key.   The used hard key determining means can accept the operation accepting means while the operation screen is displayed on the display means among one or more hard keys excluding the power key among the plurality of hard keys. The remote control system according to claim 1, further comprising a screen unit determining unit that determines the hard key to be used as a hard key to be used. The image processing device executes an application program and determines a screen for displaying an operation screen to be displayed on the display unit;
Remote control means for transmitting the operation screen to the remote operation device and causing the process execution means to execute an application program specified by remote operation received from the remote operation device among the plurality of application programs, The remote control system according to claim 1, further comprising:   The remote control system according to claim 4, wherein the used hard key determination unit includes a program analysis unit that analyzes an application program executed by the process execution unit.   The remote control system according to claim 4, wherein the use hard key determination unit includes a language analysis unit that analyzes the operation screen when the operation screen generated by the process execution unit is described in a markup language. . The image processing apparatus further includes server remote operation means for displaying an operation screen received from an external server on the display means, and transmitting a remote operation accepted by the operation accepting means to the server,
The remote control system according to claim 1, wherein the specifying unit specifies an application program that the server remote operation unit causes the external server to execute.   The remote control system according to claim 7, wherein the used hard key determination unit includes a hard key information acquisition unit that acquires information on a hard key from the external server.   2. The use hard key determination unit determines all of the plurality of hard keys except a power key as use hard keys when the application program specified by the specifying unit is a predetermined specific program. The remote control system according to any one of -8. An image processing device remotely operated by a remote operation device,
An operation receiving means having a plurality of hard keys;
Display means for displaying an operation screen described in a markup language;
Use hard key determination means for determining one or more use hard keys used on the operation screen among the plurality of hard keys by analyzing the displayed operation screen;
A hard key for generating a hard key screen in which one or more soft key images corresponding to each of the one or more used hard keys are arranged in a markup language based on the determined hard key information about the one or more used hard keys Screen generation means;
Remote operation screen generating means for generating a remote operation screen including the operation screen and the generated hard key screen;
An image processing apparatus comprising: remote control means for transmitting the generated remote operation screen to the remote operation apparatus and executing processing according to the remote operation received from the remote operation apparatus. Processing execution means for executing an application program and generating an operation screen for display on the display means in a markup language;
The image processing apparatus according to claim 10, wherein the remote control unit causes the processing execution unit to execute an application program specified by a remote operation received from the remote operation device among a plurality of application programs. The process execution means further includes server remote operation means for receiving an operation screen described in a markup language received from an external server and transmitting the server remote operation received by the operation reception means to the server. ,
When the operation screen is received by the server remote operation means, the remote control means transmits a remote operation screen including the received operation screen to the remote operation device and then receives the remote operation screen from the remote operation device. The image processing apparatus according to claim 11, wherein an operation is transmitted to the server remote operation unit as a server remote operation.   The image processing apparatus according to claim 12, wherein the used hard key determination unit includes a hard key information acquisition unit that acquires information about a hard key from the external server.   The image processing apparatus according to claim 12 or 13, wherein the used hard key information acquisition unit includes an application unit acquisition unit that receives information about the hard key before receiving the operation screen from the external server. .   The image processing apparatus according to claim 12, wherein the use hard key determination unit includes a screen unit acquisition unit that receives information on the hard key every time the operation screen is received from the external server. A remote control method executed by an image processing device and a remote operation device for remotely operating the image processing device,
The image processing apparatus includes an operation receiving unit having a plurality of hard keys;
Display means for displaying an operation screen,
A specifying step of specifying an application program corresponding to an operation screen displayed on the display unit from a plurality of application programs in the image processing apparatus;
A use hard key determination step of determining one or more hard keys defined by the application program specified in the specific step among the plurality of hard keys as the use hard key;
One or more soft key images corresponding to each of the one or more used hard keys are displayed on either the image processing device or the remote control device based on the determined hard key information about the one or more used hard keys. A hard key screen generation step for generating the arranged hard key screen;
A remote control method for executing a remote operation screen generation step of generating a remote operation screen including the operation screen and the generated hard key screen. A remote control method executed by an image processing device remotely operated by a remote operation device,
The image processing apparatus includes an operation receiving unit having a plurality of hard keys;
Display means for displaying an operation screen described in a markup language,
A used hard key determination step of determining one or more used hard keys used on the operation screen among the plurality of hard keys by analyzing the displayed operation screen;
A hard key for generating a hard key screen in which one or more soft key images corresponding to each of the one or more used hard keys are arranged in a markup language based on the determined hard key information about the one or more used hard keys A screen generation step;
A remote operation screen generating step for generating a remote operation screen including the operation screen and the generated hard key screen;
A remote control step of transmitting the generated remote operation screen to the remote operation device and executing a process according to a remote operation received from the remote operation device. A remote control program executed by a computer for controlling an image processing device remotely operated by a remote operation device,
The image processing apparatus includes an operation receiving unit having a plurality of hard keys;
Display means for displaying an operation screen described in a markup language,
A used hard key determination step of determining one or more used hard keys used on the operation screen among the plurality of hard keys by analyzing the displayed operation screen;
A hard key for generating a hard key screen in which one or more soft key images corresponding to each of the one or more used hard keys are arranged in a markup language based on the determined hard key information about the one or more used hard keys A screen generation step;
A remote operation screen generating step for generating a remote operation screen including the operation screen and the generated hard key screen;
A remote control program that causes the computer to execute a remote control step of transmitting the generated remote operation screen to the remote operation device and executing processing according to the remote operation received from the remote operation device.

Images