JP6015319B2 - Image forming system, control method thereof, and control program thereof - Google Patents

Description

  The present invention relates to an image forming system, a control method thereof, and a control program thereof, and more particularly to an image forming system capable of assisting an operation of an image forming apparatus, a control method thereof, and a control program thereof.

Conventionally, various techniques for assisting the operation of the image forming apparatus have been proposed.
For example, in Patent Document 1 (Japanese Patent Laid-Open No. 2006-030359), in an image forming apparatus in which an operation information label is attached to a place where a user needs to operate, an infrared sensor is provided on the label, and the user It is disclosed that voice guidance is played when the user approaches or touches.

  Further, in Patent Document 2 (Japanese Patent Laid-Open No. 2010-165106), when an error occurs in a multifunction device, the multifunction device sends error information to the portable terminal that has transmitted a print instruction to the multifunction device. It is disclosed that error information is displayed on the portable terminal. Note that this document also discloses that the distance between the multifunction device and the portable terminal is measured, and error information is not transmitted if the measured distance is within a predetermined distance.

JP 2006-030359 A JP 2010-165106 A

  In the image forming apparatus, when a user performs an operation such as when a guidance display is performed, an operation different from that indicated by the guidance display is performed, so that the paper is damaged or jammed (paper jam). May occur and the work efficiency of the user may be reduced.

  For example, if a jam occurs during continuous paper feeding and the paper is bitten near the paper feed unit (paper feed roller), the user does not follow the paper jam removal procedure displayed on the operation panel of the image forming apparatus. When removal (jamming) is performed, the following case 1) is assumed to occur.

Case 1)
A jam occurs in the paper feed unit, and a jam clearing procedure is displayed on the operation panel. In this procedure, an instruction is given to open the door panel that covers the paper conveyance path. However, the user who confirms the instruction opens the sheet cassette by mistake. As a result, the sheet that has been bitten by the sheet feeding unit is broken, and so-called cutting occurs. By removing the paper damaged by the cutting, the user's work efficiency is lowered.

  In addition, when toner empty occurs during printing, the following case 2) is assumed if the user tries to operate with a procedure different from the procedure for replacing the toner bottle displayed on the operation panel of the image forming apparatus. The

Case 2)
When it is detected that toner empty has occurred, a procedure for replacing the toner bottle is displayed on the operation panel. In this procedure, an instruction to open a TB door (door that opens and closes a portion in which the toner bottle is stored) in the image forming apparatus is given. However, a user who confirms the instruction erroneously opens the door panel that covers the paper conveyance path. As a result, the drive relay relating to the sheet conveyance is cut off, the sheet conveyance operation is stopped, and a jam of the sheet being conveyed occurs. The removal of the paper reduces the user's work efficiency.

  According to Patent Document 2, the user is notified that the user needs to operate the image forming apparatus when an error occurs in the image forming apparatus such as a multifunction peripheral. However, the user's erroneous operation as described above cannot be prevented simply by notifying the user.

  According to Patent Document 1, when a user's hand approaches a place that should not be touched on the image forming apparatus during operation, this can be notified. However, even if a notification is made when approaching, the notification may not be in time.

  The present invention has been conceived in view of such circumstances, and an object thereof is to reliably prevent an erroneous operation of a user in an image forming apparatus.

  An image forming system according to the present invention is an image forming system including an image forming apparatus and a terminal device capable of communicating with the image forming apparatus. The image forming apparatus includes a plurality of operation units and includes a plurality of operation units. Procedure information of the operation when a state requiring a user operation on any of the operation units occurs, and prohibited operation unit information that is information of an operation unit that is prohibited from being operated by the user when the state occurs The terminal device includes a camera and a control unit for controlling the image forming apparatus, and outputs a warning to the user of the terminal device and a detection unit for detecting the position of the terminal device with respect to the image forming device When a warning occurs and when a situation occurs, the user's action is monitored based on the camera's shooting result, and when it is detected that the user has acted on the operation part of the prohibited operation part information And an output instruction unit for outputting a warning alert means.

  Preferably, the output instructing unit monitors the user's operation based on the photographing result of the camera on the condition that the position detected by the detecting unit is within a predetermined range from the position of the prohibited operation unit information.

  Preferably, the output instruction unit warns the warning unit when it is detected that the line of sight of the person detected from the photographing result of the camera is continuously directed to the operation unit of the prohibited operation unit information for a predetermined time. Is output.

  Preferably, the output instruction means causes the warning means to output a warning when it is detected from the photographing result of the camera that the movement of the person is a movement toward the operation section of the prohibited operation section information.

  Preferably, the output instruction unit causes the warning unit to output a warning when the speed of the movement toward the operation unit is equal to or higher than a predetermined speed by using a speed measurement mechanism capable of measuring the speed of the user in the moving direction.

Preferably, the predetermined range is a predetermined range which is assumed to be operable a plurality of operation unit manually.

  Preferably, when the output instruction means detects that the user has acted on an operation unit other than the operation unit that requires user operation and the operation unit of the prohibited operation unit information among the plurality of operation units. In addition, it is notified that an operation unit different from the operation unit indicated by the procedure information is about to be operated.

  Preferably, the operation unit whose operation is prohibited by the prohibited operation unit information is held for each state, and the predetermined time is set for each state.

  Preferably, an infrared sensor for detecting the positional relationship between the image forming apparatus and the object is further provided, and the detection unit corrects the position of the terminal device with respect to the image forming apparatus based on the detection output of the infrared sensor.

Preferably, the terminal device is movable away from the image forming apparatus.
An image forming system control method according to the present invention is a control method executed by a computer that controls an image forming system including an image forming apparatus and a terminal device capable of communicating with the image forming apparatus. The apparatus has a plurality of operation units, and the procedure information of the operation when a state requiring the user's operation on any of the plurality of operation units occurs and the operation by the user when the state occurs. The prohibited operation unit information, which is information of the prohibited operation unit, is held, and the terminal device includes a camera and a control unit for controlling the image forming device, and detects the position of the terminal device with respect to the image forming device. A step, a step of outputting a warning to the user of the terminal device, and when a situation occurs, the user's operation is monitored based on the camera's shooting result, and the prohibited operation unit information When the user detects that the mounted an operation to create unit, and a step of outputting a warning.

  An image forming system control program according to the present invention is a control program executed by a computer that controls an image forming system including an image forming apparatus and a terminal device capable of communicating with the image forming apparatus. The apparatus has a plurality of operation units, and the procedure information of the operation when a state requiring the user's operation on any of the plurality of operation units occurs and the operation by the user when the state occurs. Forbidden operation unit information, which is information of the prohibited operation unit, the terminal device includes a camera and a control means for controlling the image forming apparatus, and the control program stores the image of the terminal device on the computer. A step of detecting the position relative to the forming device, a step of outputting a warning to the user of the terminal device, and if a situation occurs, And Zui monitors the user's operation, when the user detects that the mounted an operation to the operation unit of the prohibition operation unit information, and a step of outputting the warning.

  According to the present invention, the operation of the user is monitored based on the result of shooting by the camera. As a result of the monitoring, a warning is output when a user action on the operation unit of the prohibited operation unit information is detected.

  As a result, it is possible to detect the user's operation at a position further away from the image forming apparatus, to detect that the user has operated the operation unit of the prohibited operation unit information at an early stage, and to output a warning. it can.

  Therefore, in the image forming apparatus, it is possible to reliably prevent a user's erroneous operation.

1 is a schematic front sectional view showing an MFP (Multi-Functional Peripheral) 1 which is an embodiment of an image forming system of the present invention. 2 is a control block diagram of an MFP. FIG. It is a figure which shows an example of the content of the position information for control typically. It is a figure for demonstrating the coordinate of FIG. 10 is a flowchart of processing executed when a jam occurs in the first paper feed cassette. It is a figure for demonstrating an example of a camera monitoring effective range. It is a flowchart of the process performed in order to pinpoint the position of an operating terminal. It is a figure for demonstrating the processing content of FIG. It is a flowchart of an example of the processing content of judgment in step S90 of FIG. It is a figure for demonstrating the processing content of FIG. It is a figure for demonstrating the processing content of FIG. It is a figure for demonstrating the processing content of FIG. It is a flowchart of the process which detects imaging | photography direction (theta) now of a camera. It is a figure for demonstrating the processing content of FIG. It is a figure which shows an example of the external appearance of a wristband type operation terminal. It is a flowchart of an example of the processing content using the operation terminal of FIG.

  Hereinafter, an image forming system according to the present invention will be described with reference to the drawings. In addition, in each figure, the same code | symbol is attached | subjected about the component which show | plays the same effect | action and function, and the description is not repeated.

[Overall configuration of image forming system]
FIG. 1 is a schematic front sectional view showing an MFP (Multi-Functional Peripheral) 1 which is an embodiment of an image forming system of the present invention. The MFP 1 is a multifunction machine having various image processing functions such as a print function, a facsimile function, a copy function, and a scanner function.

  As illustrated in FIG. 1, the MFP 1 includes an image reading unit 10, a printer unit 20, an operation terminal 300, and a controller 200. The controller 200 controls the overall operation of the MFP 1.

  Operation terminal 300 is operated by a user to input an instruction to MFP 1. The operation terminal 300 is provided with a camera 307 and a panel display unit 309.

  In the MFP 1, a place where the operation terminal 300 is installed is provided as an installation place 811. In the present embodiment, operation terminal 300 is configured to be movable with respect to the MFP 1 main body. In MFP 1, operation terminal 300 is basically placed at installation location 811. The user can take the operation terminal 300 out of the installation location 811 in order to confirm the display on the panel display unit 309.

  In the present embodiment, panel display unit 309 is realized by a touch panel that accepts a touch operation from a user. In the image forming system, the display unit is not necessarily realized by a touch panel. The display unit and the operation unit (a part that receives an operation from the user) may be realized separately.

  An imaging unit 150K, 150Y, 150M, 150C and ink cartridges 151K, 151Y, 151M, 151C are detachably mounted on the printer unit 20. The imaging units 150K, 150Y, 150M, and 150C and the ink cartridges 151K, 151Y, 151M, and 151C are covered by a door (toner replacement door 1A in FIG. 4) (not shown) when mounted on the MFP 1, It is configured to be included in.

  The image reading unit 10 includes a scanner, and is a known device that reads an image of a document placed on a document glass plate (not shown) by moving the scanner. In the image reading unit 10, an original image obtained by irradiation with an exposure lamp is formed by a condenser lens, and further, three different wavelengths of red (R), green (G), and blue (B) are collected by a spectroscope. The light is split into light and respectively incident on a red CCD (Charge Coupled Device) image sensor, a green CCD image sensor, and a blue CCD image sensor. An output signal from each CCD image sensor (hereinafter simply referred to as “CCD sensor”) is AD (analog-to-digital) converted by the controller 200 and converted into R, G, B image data of the document.

  The obtained image data for each color component is subjected to various types of data processing in the controller 200, and further each reproduction color of black (K), yellow (Y), magenta (M), and cyan (C) (hereinafter referred to as “color”). These K, Y, M, and C are added as subscripts to the component numbers associated with each reproduction color. The converted image data is stored for each reproduction color in a RAM (RAM (Random Access Memory) 201 described later) in the controller 200 and subjected to various correction processes such as registration correction. Thereafter, the image data is read for each scanning line in synchronism with the supply of printing paper (hereinafter simply referred to as “paper”), and the laser diode for exposing the photosensitive drums 51K, 51Y, 51M, 51C is driven. Signal.

  The printer unit 20 includes a transfer belt 41 that is stretched and driven in the printer unit 20, and imaging units (image forming units) 150K, 150Y, 150M, which are arranged at predetermined intervals so as to face the transfer belt 41. 150C, exposure scanning units 60K to 60C provided for each imaging unit, a sheet feeding unit 70 for feeding a sheet to the transfer belt 41, a fixing unit 80 disposed on the downstream side of the sheet conveying unit 40, and both surfaces Unit 82.

  The exposure scanning units 60K to 60C each perform exposure scanning in the main scanning direction on the photoconductive drums 51K to 51C by receiving a driving signal output from the controller 200 and emitting laser light, or deflecting the laser light. For example, a polygon mirror (not shown) is provided.

  The paper supply unit 70 is configured to supply paper cassettes 71 and 72 for storing paper, pick-up rollers 76A to 76C for feeding out the paper stored in each paper feed cassette, and timing for feeding the fed paper to the transfer belt 41. A registration roller 79 for adjustment is included.

  In this specification, for convenience, the paper feed cassette 71 may be referred to as a “first paper feed cassette” and the paper feed cassette 72 may be referred to as a “second paper feed cassette”.

  The sheet conveyance path in the vicinity of the registration roller 79 is covered with a front door (not shown) (front door 1B in FIG. 4).

  Sheets of different sizes or orientations are stored in the sheet feeding cassettes 71 and 72. Specifically, for example, A5 size (horizontal feed) paper (hereinafter referred to as “A4 paper”) is stored in the paper feed cassette 71, and A4 size (horizontal feed) paper (in the paper feed cassette 72). (Hereinafter referred to as “B4 paper”).

  The paper stored in the paper feed cassette 71 is fed out by the pickup roller 76A, detected by the paper sensor 911, and then sent onto the transport path.

  The paper stored in the paper feed cassette 72 is fed out by the pickup roller 76B, detected by the paper sensor 912, and then sent onto the transport path.

  On the conveyance path, a paper sensor 915 that detects the presence or absence of paper at the junction of the paper fed from the paper feed cassette 72 and the paper fed from the paper feed cassette 71 is provided. The paper that has passed the position detected by the paper sensor 915 is sent out to the transfer belt 41 by the registration roller 79.

  The paper on which the toner images of the respective colors are multiplex-transferred by the transfer belt 41 is detected by the paper sensor 916 and then conveyed to the fixing unit 80. The fixing roller 801 of the fixing unit 80 includes a heater 917 inside. The controller 200 controls energization to the heater 917 while detecting the surface temperature of the fixing roller 801 with a temperature detection sensor. The sheet is pressurized at a high temperature by the fixing roller 801, and toner particles on the surface thereof are fused and fixed on the sheet surface. After the sheet sensor 920 detects the sheet, the sheet is discharged to the sheet discharge tray 91.

  In the MFP 1, when an image is formed on the back side in addition to the front side of the sheet, the image formed on the front side of the sheet is fixed by the fixing unit 80, and then the sheet is sent to the duplex unit 82.

  Paper sensors 919 and 918 are sequentially provided on the paper transport path in the duplex unit 82. When the sheet is sent again from the duplex unit 82 to the transfer belt 41, an image is formed on the back surface of the sheet in the same manner as the image forming apparatus is formed on the front surface. Then, after the image formed on the back surface is fixed by the fixing unit 80, the paper is detected by the paper sensor 920 and discharged to the paper discharge tray 91.

  On the side surface of the MFP 1 on the duplex unit 82 side, a door (not shown) (a vertical conveyance door described later) is provided. The vertical conveyance door is opened and closed in order to remove from the outside the paper that has been caught in the conveyance path by a jam in the duplex unit 82.

  The MFP 1 is provided with a manual feed tray 95. In the MFP 1, the paper set on the manual feed tray 95 may be a target for image formation. The paper set on the manual feed tray 95 is sent to the transport path, detected by the paper sensor 915, and then sent to the transfer belt 41 to form an image.

  In the MFP 1, when an appropriate paper sensor does not detect the passage of paper at a timing corresponding to the paper conveyance timing, or when the paper sensor continuously detects a paper for a period longer than the period assumed by the conveyance timing, It is determined that a jam has occurred.

[Block configuration]
FIG. 2 is a control block diagram of the MFP 1.

  In FIG. 2, the MFP 1 is divided into an image forming unit 100, a controller 200, and an operation terminal 300.

  Image forming unit 100 is a part in MFP 1 where an image forming operation, an operation for paper conveyance, and the like are performed.

  The controller 200 instructs the operation of the image forming unit 100 in response to an instruction input from the operation terminal 300 or the like. In addition, when the image forming unit 100 enters a state that requires a user's work such as a paper jam, the controller 200 displays an operation procedure for canceling the state on the operation terminal 300. Let

Hereinafter, the configuration of each unit will be described.
(Image forming unit 100)
The image forming unit 100 includes an engine control unit 102, an I / F (interface) unit 104, a motor drive unit 109, a ROM (Read Only Memory) 105, and a RAM (Random Access Memory) centered on an engine CPU (Central Processing Unit) 103. ) 106 is present.

  In the image forming unit 100, the paper feed motor of the motor drive unit 109 is driven to feed paper to the paper image transport path stored in each cassette, and the timing motor is driven to feed paper. Paper is transported.

  The engine control unit 102 includes a timing sensor 110 (paper sensor 915) and a paper feed sensor 101 (paper sensors 911 and 912 in FIG. 1). The engine CPU 103 detects whether or not a jam has occurred in each paper feed cassette based on the detection outputs of these sensors. Note that detection of the occurrence of a jam based on the detection output of the paper sensor on the conveyance path can be realized based on a known technique, and thus detailed description thereof will not be repeated here.

  The engine CPU 103 communicates with the controller 200 (the I / F unit 207) via the I / F unit 104. When the occurrence of a jam is detected, the engine CPU 103 transmits information about the jam (jam position, etc.) to the controller 200.

(Controller 200)
The controller 200 includes an image processing unit 206, a terminal tracking control unit 204, a camera image processing unit 205, an I / F unit 207, and a memory unit including a RAM 201 and a ROM 202, centering on the controller CPU 203.

  The controller CPU 203 acquires information such as the position information of the camera 307 and the shooting direction from the operation terminal 300 via the I / F unit 207. Then, the controller CPU 203 sends these pieces of information to the terminal tracking control unit 204. The terminal tracking control unit 204 tracks the position of the operation terminal 300 based on the information sent from the controller CPU 203.

  Further, the controller CPU 203 acquires an image captured by the camera 307 via the I / F unit 207 and sends the image to the camera image processing unit 205. The camera image processing unit 205 performs image processing on the sent image and transmits the result to the controller CPU 203.

  The image processing unit 206 performs various image processing such as γ correction on the image read by the scanner and the image accumulated in the image forming unit 100. In the MFP 1, the image after the image processing may be an object of image formation on a sheet in the image forming unit 100.

  It should be noted that elements (terminal tracking control unit 204 and camera image processing unit 205) for tracking the position and the like of operation terminal 300 in controller 200 can be connected to a device different from MFP 1 (for example, MFP 1 via a network). Server). In this case, the server can communicate with the operation terminal 300, and information for specifying the position of the operation terminal 300 based on information acquired from the operation terminal 300 (such as a detection output of a three-dimensional acceleration sensor 303 described later). Is transmitted to the controller 200. The controller 200 acquires information about the position of the operation terminal 300 based on information from the server.

  In the present embodiment, the MFP 1 realizes at least a part of the functions described in the present specification when the controller CPU 203 executes a program stored in the ROM 202 or the like. Note that the program may be recorded on a recording medium fixed to the main body of the MFP 1 or may be recorded on a recording medium detachable from the main body. Recording media include CD-ROM (Compact Disc-Read Only Memory), DVD-ROM (Digital Versatile Disk-Read Only Memory), USB (Universal Serial Bus) memory, memory card, FD (Flexible Disk), hard disk, magnetic Tape, cassette tape, MO (Magnetic Optical Disc), MD (Mini Disc), IC (Integrated Circuit) card (excluding memory cards), optical card, mask ROM, EPROM, EEPROM (Electronically Erasable Programmable Read-Only Memory), etc. And a medium for storing the program in a nonvolatile manner. The program executed by the controller CPU 203 may be installed in the MFP 1 via a network.

(Operation terminal 300)
The operation terminal 300 includes a memory unit including an operation unit 301, a panel display unit 309, a gyro sensor 302, a three-dimensional acceleration sensor 303, an I / F unit 308, a ROM 305, and a RAM 306, centering on the operation terminal CPU 304.

  In the present embodiment, operation terminal 300 is configured to be movable with respect to the MFP 1 main body (a housing that houses at least part of image forming unit 100 and controller 200). The operation terminal 300 is provided with a gyro sensor 302 that detects an angular velocity when the operation terminal 300 moves, and a three-dimensional acceleration sensor 303 that detects acceleration in three axial directions applied to the operation terminal 300. The operation terminal CPU 304 transmits these detection outputs to the controller 200 via the I / F unit 308.

  In the operation terminal 300, the camera 307 is fixed to the casing of the operation terminal 300. In the present embodiment, the position of the operation terminal 300 is specified based on the detection output of the three-dimensional acceleration sensor 303, and the shooting direction of the camera 307 is specified based on the detection output of the gyro sensor 302.

  Further, when the controller CPU 203 determines that a warning is given to the user, the warning information coming from the I / F unit 207 is received by the I / F unit 308 and displayed on the panel display unit 309.

[Position information for control]
In the MFP 1, a position where an operation by the user is required or a position where the operation by the user is prohibited is set when jamming or the like requires an operation by the user. Further, the MFP 1 detects a position that is predicted to be a target of an operation by the user. If the detected position is a position where the operation is prohibited, a warning is output to the user.

  In order to output such a warning, the MFP 1 registers position information of a portion that can be operated. In this specification, a portion that can be an operation target in the MFP 1 is referred to as an “operation unit”. An example of the registered position information will be described below as control position information.

FIG. 3 is a diagram schematically illustrating an example of the content of the control position information.
The control position information as shown in FIG. 3 is stored in the ROM 202. The ROM 202 stores control position information for each state that requires user operation. The control position information shown in FIG. 3 is information when a jam occurs in the first paper feed cassette (paper feed cassette 71).

In FIG. 3, a list of coordinate data for each operation unit is shown as the control position information.
In the list, for each operation unit, a name (operation unit name), a coordinate data group, user operation prohibition information, user operation promotion information, a user gaze time threshold value, an operation determination speed threshold value, and an operation determination time A threshold is set.

  The operation unit names are “portable terminal installation location” “front door operation location” “toner replacement door operation location” “vertical transfer door operation location” “first paper feed cassette operation location” “second paper feed cassette operation location” Is listed.

  The “portable terminal installation location” is an installation location 811 where the operation terminal 300 is installed. “Front door operation location”, “Toner replacement door operation location”, and “Vertical transport door operation location” are the parts operated when opening the front door, toner replacement door, and vertical transport door. is there. Note that the front door and the toner replacement door are shown as a toner replacement door 1A and a front door 1B, respectively, in FIG. The “first paper feed cassette operation location” and “second paper feed cassette operation location” are portions (handle portions and the like) that are operated when the paper feed cassettes 71 and 72 are opened.

  The coordinate data group is the coordinates of the position to be operated by each operation unit. In this embodiment, the origin of the coordinates is the center of the installation location 811 as shown in FIG. Specifically, the origin of the coordinates is a point where the center of the operation terminal 300 is located when the operation terminal 300 is installed at the installation location 811. The unit of coordinates in FIG. 3 is 1 cm. Note that the coordinates and units in FIG. 3 are merely examples, and the present invention is not intended to be limited to these numerical values.

  The “coordinate data group” of the control position information is the coordinates of the end portions (for example, the upper left end, the upper right end, the lower left end, and the lower right end) of each operation location. That is, the coordinate data group specifies the range of each operation location.

  The user operation prohibition information is information indicating whether or not an operation on each operation location is prohibited in this state. In the example of FIG. 3, only the operation location of the first sheet cassette is “valid”, and the other operation locations are “invalid”. This corresponds to FIG. 3 that, when a jam occurs in the first paper feed cassette, the operation is prohibited for the first paper feed cassette, and the operation is not prohibited for the other operation locations. Show.

  In the present embodiment, “operation is prohibited” means that a warning is output when an operation is predicted to be performed on the operation location. The operation is prohibited only when a state corresponding to each position information for control occurs. That is, according to FIG. 3, the operation on the first paper feed cassette is prohibited when a jam occurs in the first paper feed cassette corresponding to FIG. When the jam is released, the prohibition of the operation on the first paper feed cassette based on FIG. 3 is also released.

  The user operation promotion information is information indicating that the user operation is promoted in the state. In the example of FIG. 3, only the vertical transfer door operation location is “valid”, and the other operation locations are “invalid”. This means that in the state where a jam has occurred in the first paper feed cassette corresponding to FIG. 3, the MFP 1 prompts the user to operate the vertical conveyance door and does not prompt other operations. Is shown.

  The user gazing time threshold is a threshold for determining that the user is gazing at the operation location. In the example of FIG. 3, for example, “1s” is set for the front door. In MFP 1, when it is determined that the user is continuously looking at the operation position of the front door for one second or more, it is determined that the user is paying attention (gazing) to the operation position of the front door. Means that.

  In the present embodiment, “gazing” means a state in which the line of sight is continuously directed to a specific location for a predetermined time or more.

  The operation determination speed threshold is a threshold used in the process of FIG. 16 and is a speed threshold for determining that the user is moving toward the operation location. In the example of FIG. 3, for example, the front door is set to “15 km / h”. This means that in the MFP 1, when it is detected that the front door exists ahead of the movement direction of the user and the movement speed is 15 km / h or more, the user moves to the operation position of the front door. It means to judge that it is moving toward.

  The operation determination time threshold is a threshold used in the processing of FIG. 16 and is a threshold for determining that the user is trying to perform an operation on the operation location. In the example of FIG. 3, for example, “100 ms” is set for the front door. This means that in the MFP 1, when it is determined that the user continuously moves the hand toward the front door for 100 milliseconds or more, it is determined that the user is about to operate the operation portion of the front door. means.

  The information constituting the control position information as shown in FIG. 3 is assumed to be a different combination for each state that requires user operation. Such a state includes replacement of a toner bottle and out of paper in addition to the paper jam as shown in FIG.

[Warning output processing]
Next, processing contents when a state in which the user needs to operate the operation unit in the MFP 1 occur will be described.

  FIG. 5 is a flowchart of processing executed by the controller CPU 203 when a jam occurs in the first paper feed cassette.

  Referring to FIG. 5, when controller CPU 203 detects the occurrence of a jam in the first paper feed cassette, in step S10, controller CPU 203 causes operation terminal 300 to display a procedure for releasing the jam. The occurrence of a jam in the MFP 1 is detected based on, for example, the detection output of various paper sensors, and the type (occurrence location) of the jam is specified based on the detection output of various paper sensors. The display of the procedure for releasing the jam is realized by transmitting information stored in the ROM 202 in association with the identified jam type to the operation terminal 300. Information for displaying the above procedure is stored in the ROM 202 in association with the type of jam that has occurred. Since the above-described identification of the type and display of the procedure related to the type can be realized by a known technique, detailed description will not be repeated here.

  Next, the controller CPU 203 determines whether or not the operation terminal 300 has been removed from the installation location 811 in step S20. This determination is realized based on whether or not the position of the operation terminal 300 derived based on the detection output of the three-dimensional acceleration sensor 303 is different from the position where the operation terminal 300 should be installed. If it is determined that it has been removed, the process proceeds to step S30.

  In the present embodiment, it is assumed that operation terminal 300 is installed at installation location 811 when power to MFP 1 is turned on. In the MFP 1, the controller CPU 203 specifies the current position of the operation terminal 300 with reference to the position of the operation terminal 300 when the power is turned on. The position of the operation terminal 300 is derived based on the detection output of the three-dimensional acceleration sensor 303.

  In step S30, the controller CPU 203 specifies the position of the operation terminal 300 at that time.

  Next, in step S40, the controller CPU 203 determines whether or not the operation terminal 300 is located within the camera monitoring effective range. The camera monitoring effective range is a range in which it is assumed that the MFP 1 can be operated manually. For example, as shown by a range R01 in FIG. 6, the range is within a distance of 1.5 m from the MFP 1. When controller CPU 203 determines that operation terminal 300 is within the range, controller CPU 203 proceeds to step S70. When controller CPU 203 determines that operation terminal 300 is outside the range, controller CPU 203 proceeds to step S50.

  In step S50, controller CPU203 advances a process to step S60, without performing a special process.

  In step S60, the controller CPU 203 determines whether or not a predetermined control period T (for example, 5 ms) has elapsed since the execution of the process of the previous step S30. Return processing.

  On the other hand, in step S <b> 70, the controller CPU 203 uses the camera 307 of the operation terminal 300 to start monitoring user operations.

  In step S80, the controller CPU 203 detects the user's line of sight and movement as a result of the monitoring in step S70, and proceeds to step S90.

  The monitoring of the user's operation at step S70 and the detection of the line of sight and the operation at step S80 will be described later.

  In step S90, based on the detection result in step S80, the controller CPU 203 determines whether or not the user's line of sight or action is directed to the operation unit specified by the user operation prohibition information in the control position information. If it is determined that it is directed, the process proceeds to step S100, and if it is determined that it is not directed, the process proceeds to step S50.

  In step S100, the controller CPU 203 outputs a warning to the user and ends the process.

  The warning output here is realized by, for example, displaying a warning message on the panel display unit 309 of the operation terminal 300. The warning is a warning that the user is trying to operate an operation unit that is prohibited from being operated. For example, in the example of FIG. 3, when a jam has occurred in the first paper feed cassette, an operation of opening the first paper feed cassette by operating the operation position of the first paper feed cassette is prohibited. According to this, when a jam occurs in the first paper feed cassette and the user who has the operation terminal 300 starts to open the first paper feed cassette, the panel display unit 309 is displayed. Displays a warning message.

  The warning message may be a notification that the operation to the operation unit is prohibited, such as “Operation to the first sheet cassette is currently prohibited.” It is good, and the operation part to be prohibited and the reason why it is prohibited, such as “There is a possibility that paper will be cut when the first paper cassette is opened because a jam has occurred in the first paper cassette.” May be used. In addition, there is a possibility that paper will be cut if the first paper cassette is opened because a jam has occurred in the first paper cassette. Please open the vertical transfer door. A message for guiding the operation unit specified by the promotion information to be operated may be included.

[Detection of operation terminal position]
FIG. 7 is a flowchart of processing executed for specifying the position of the operation terminal 300.

  As described with reference to FIG. 5, the controller CPU 203 detects the position of the operation terminal 300 in step S <b> 30 every control cycle T until a warning is output in step S <b> 100. Hereinafter, the specification of the position of the operation terminal 300 in step S30 will be described.

  Referring to FIG. 7, in step S301, controller CPU 203 sets the initial position coordinate of operation terminal 300 to the current terminal position coordinate Now. The initial position coordinates are position coordinates when the operation terminal 300 is installed at the installation location 811.

  Next, the controller CPU 203 detects the magnitude and direction of the acceleration in each of the three-dimensional directions detected by the three-dimensional acceleration sensor 303 in step S303.

  Next, in step S305, the controller CPU 203 determines the acceleration magnitude and elapsed time in the three-dimensional direction detected in step S303 (the elapsed time from the previous position detection, which is the above-described control cycle T). The movement distance A01 of the operation terminal 300 is calculated, and in step S307, the movement direction is calculated.

  Specifically, as shown in FIG. 8, when the acceleration in the x direction is Ax, the acceleration in the y direction is Ay, and the acceleration in the z direction is Ax, the acceleration obtained as a sum when these are used as vectors. The direction of a (arrow A01 in FIG. 8) is the moving direction in step S307, and the distance obtained when moving by the control time T with the acceleration represented by the moving direction is the moving distance in step S305.

  Returning to FIG. 7, after step S <b> 307, the controller CPU 203 determines in step S <b> 309 the current position coordinates of the operation terminal 300 based on the movement distance and direction obtained in steps S <b> 305 and S <b> 307 and the terminal position coordinate Now. Anew is calculated.

  In step S311, the controller CPU 203 updates the terminal position coordinate Anow to the current position coordinate Anew calculated in step S309, and ends the process.

  Thereby, the controller CPU 203 calculates the position of the operation terminal 300 for each control period T based on the detection output of the three-dimensional acceleration sensor 303.

[Detection of user's eyes]
In step S90, it is determined whether or not the user's line of sight or action is directed to the operation unit specified by the user operation prohibition information in the control position information. The contents of this process will be described below.

FIG. 9 is a flowchart of the processing content of the determination in step S90.
Referring to FIG. 9, in step S <b> 901, controller CPU 203 extracts the contours of both eyes and nose using a known image processing technique for the image captured by camera 307.

  Next, in step S903, as shown in FIG. 10, the controller CPU 203 adds a deviation (angle) from the camera 307 to the position where the eyes and nose are located with respect to the current camera shooting direction θnow. The direction to the position where the nose is located is detected. In FIG. 10, together with the shooting direction θnow of the camera 307, the direction from the camera 307 toward the left eye is indicated by “Rayel θnow”, the direction from the camera 307 toward the right eye is indicated by “Leyer θnow”, and directed from the camera 307 toward the nose. The direction is indicated by “Lnose θnow”.

  A gyro sensor 302 is used to detect the shooting direction θnow of the camera 307. This detection method will be described later with reference to FIG.

  Detection of the deviation between both eyes and nose is calculated based on how much the positions of both eyes and nose are deviated from the center of the image when the center of the captured image is θnow.

  In step S905, the controller CPU 203 calculates the distances between the eyes and the nose from the camera 307 by using the auto force function of the camera.

  Next, in step S907, the controller CPU 203 determines, based on the position coordinates of the current operation terminal 300 (camera 307), the direction and distance of both eyes and nose from the camera 307 calculated in step S905. Specify the coordinate position of the nose.

  Next, in step S909, the controller CPU 203 calculates a straight line representing the user's line of sight (line-of-sight linear equation I) based on these coordinate positions. The calculation of the line-of-sight linear equation I will be described with reference to FIG.

  Referring to FIG. 11, first, the center of gravity G of the coordinate positions of both eyes and the nose is calculated. In FIG. 11, the coordinates of the right eye are Ert, the coordinates of the left eye are Elt, and the coordinates of the nose are N. The right eye, the left eye, and the nose mentioned here are specified by image processing on an image captured by the camera 307, respectively. The coordinates of the barycentric point G are indicated by G (Xj, Yk, Zl).

  Then, an equation of a straight line that passes through the barycentric point G and that is perpendicular to the base of the inverted triangle having the coordinate positions of both eyes and nose as a vertex is calculated as a line-of-sight linear equation I. In FIG. 11, the line-of-sight linear formula I (xyz) is indicated by a one-dot chain line.

  Returning to FIG. 9, the controller CPU 203 calculates the line-of-sight linear formula I, and then, in step S911, the coordinate data group of the operation unit specified by the user operation prohibition information on the line-of-sight linear formula I (see FIG. 3). It is determined whether or not there is a point in the area specified by. FIG. 12 is a diagram for explaining the contents of the determination. In FIG. 12, an operation location 71A indicates an area for specifying the operation location of the first paper feed cassette (paper feed cassette 71). Further, in FIG. 12, a state image in which a point on the line-of-sight linear equation I (xyz) is included in the region indicated by the operation location 71A is shown.

  Returning to FIG. 9, in step S911, the controller CPU 203 determines that there is a point in the region specified by the coordinate data group of the operation unit specified by the user operation prohibition information on the line-of-sight linear equation I, step S915. If it is determined that it does not exist, the process proceeds to step S913.

  In step S <b> 913, the controller CPU 203 clears the count value of the gaze counter Ceye on the assumption that the line of sight is away from the operation unit prohibited, and returns the process to FIG. 5. In this case, in step S90, it is determined that the user's line of sight is not directed to the operation unit prohibited from the user operation, and the process proceeds to step S50.

  On the other hand, in step S915, the controller CPU 203 increments the gaze counter Ceye by 1 and advances the process to step S917.

  In step S917, the controller CPU 203 determines whether or not the gaze counter Ceye has exceeded the gaze determination threshold value. The gaze determination threshold is a count value corresponding to a time corresponding to the user gaze time threshold in FIG. If the controller CPU 203 determines that the gaze counter Ceye has exceeded the gaze determination threshold, the controller CPU 203 advances the process to step S919.

  On the other hand, when determining that it has not exceeded yet, the controller CPU 203 returns the process to step S90 in FIG. In this case, the process proceeds from step S90 to step S50.

  In step S919, the controller CPU 203 determines that the user's line of sight is directed to the operation unit where the operation is prohibited, and returns the process to step S90 (FIG. 5). As a result, the process proceeds from step S90 to step S100.

[Detect camera direction]
A method for detecting the shooting direction θnow of the camera 307 used in step S903 in FIG. 9 will be described.

FIG. 13 is a flowchart of processing for detecting the shooting direction θnow of the camera 307.
Referring to FIG. 13, in step S9031, the controller CPU 203 sets the shooting direction of the camera 307 at the initial position of the camera 307 to the initial value of the shooting direction θnow, and advances the process to step S9033. Note that the initial position of the camera 307 is a position set in the installation location 811. The shooting direction of the camera 307 at the initial position is stored in the ROM 202, for example.

  In step S9033, the controller CPU 203 obtains the angular velocity detection result by the gyro sensor 302, and advances the process to step S9035.

  In step S9035, the controller CPU 203 calculates an angle when the controller CPU 203 rotates during the control cycle T (elapsed time T) at the angular velocity acquired in step S9033, and the process proceeds to step S3037.

  In step S3037, the controller CPU 203 calculates θnew by adding the angle obtained in step S3035 to θnow, and advances the process to step S3039.

  FIG. 14 is a diagram for explaining the calculation of θnew. In FIG. 14, the angular velocity acquired in step S3033 is indicated by angular velocity ω. FIG. 14 shows θnew obtained by adding the angle obtained in step S3035 to θnow.

  Returning to FIG. 13, in step S3039, the controller CPU 203 updates θnow by replacing θnew calculated in step S3037, and returns the process to FIG.

  In step S903 in FIG. 9, θnow updated in step S3039 is used as the shooting direction θnow.

[Summary of embodiment]
In the present embodiment described above, when a user operation such as a jam occurs, a required user operation procedure is displayed on the operation terminal 300 (step S10). In response to this, the user removes the operation terminal 300 from the installation location 811 in order to perform an operation while viewing the procedure displayed on the operation terminal 300 (step S20).

  In the MFP 1, a user who views the procedure on the operation terminal 300 is photographed by the camera 307, an image obtained by the photographing is acquired, and it is detected whether the user is not gazing at the prohibited operation unit. More specifically, it is determined whether or not the operation terminal 300 is within the camera monitoring effective range by the processing described with reference to FIGS. 5 and 9. A user's eye is detected. Then, the detected line of sight is a predetermined time (the time specified by the user gaze time threshold (FIG. 3), and the gaze counter Ceee makes a gaze determination to the operation unit in which the user's operation is prohibited in this situation. When it is detected that the user is continuously directed for a time exceeding the threshold), it is determined that the user is operating the operation unit, and a warning is output.

  Note that the determination of whether or not the operation terminal 300 is within the camera monitoring effective range corresponds to the determination of whether or not the operation terminal 300 is within a predetermined range with respect to an operation unit that is prohibited from being operated. . In the present embodiment, it is intended to output a warning when a user is likely to manually operate an operation unit that is prohibited from being operated. When the operation terminal 300 is located at a place where the operation unit cannot be operated, that is, when the user with the operation terminal 300 is present at a position where the operation unit cannot be manually operated, there is no need to issue a warning. Therefore, when the operation terminal 300 is in such a position, only tracking of the position of the operation terminal 300 is executed (step S30).

  In the present embodiment described above, a warning is output when it is detected that the user is gazing at the operation unit prohibited from the operation in the control position information of FIG. 3 (step S100). . In addition, even if the user's gaze is for an operation unit other than the operation unit for which the operation is promoted even if it is other than the operation unit for which the operation is prohibited in the control position information, that is the case. May be notified.

  That is, for example, in the example shown in FIG. 3, the front door, the toner replacement door, and the second paper feed cassette can be operated even when the operation unit is promoted (the user operation promotion information is “valid”). It is not a prohibited operation unit (user operation prohibition information is “valid”). When it is detected that any of these is the target of gaze, the controller CPU 203 may cause the panel display unit 309 to perform a notification operation in a manner different from the above warning. As a specific example of the content to be notified, please operate △△ (operation part where user operation promotion information is set to “valid”) instead of “XX (operation part detected as a target of gaze)” . Message.

  Further, in the present embodiment described above, operation terminal 300 is exemplified as being removable from MFP 1 main body (movable away from the main body). Thereby, in step S20 (refer FIG. 5), the process after step S30 was performed on condition that the operating terminal 300 was removed from the installation place 811. FIG.

  Even if the operation terminal 300 is not removed from the installation location 811, it can be detected that the user is gazing at various operation locations of the panel display unit 309 and the MFP 1 main body in the camera built in the operation terminal 300. In such a case, the process may proceed to step S30 and subsequent steps. In this case, the operation terminal 300 may be configured to be detachable from the MFP 1 main body, or may be configured to be fixed to the MFP 1 main body.

[Detection of user movement by wristband type operation terminal]
In the present embodiment described above, it is detected by the camera 307 of the operation terminal 300 whether or not the user is gazing at the operation unit prohibited from operation. The operation terminal 300 may be of a type that is worn by the user. Then, the controller CPU 203 may predict an action to be performed by the user of the operation terminal 300 by tracking the position of the operation terminal 300 and output a warning based on the prediction result.

An example of the type of operation terminal 300 to be attached is shown in FIG.
The operation terminal 300 of FIG. 15 is a wristband type, and includes a band 390 for fixing the main body of the operation terminal 300 to the wrist of the user. And the installation place 811 is comprised by the cylinder shape in which the operating terminal 300 is installed.

  The three-dimensional acceleration sensor 303 is also used to detect the position of the operation terminal 300 of this type as described with reference to FIG.

  In the example using the operation terminal 300 of FIG. 15, the process shown in FIG. 16 is executed instead of the process shown in FIG. In the following description, the operation terminal 300 may be referred to as “wristband” for convenience.

  Referring to FIG. 16, controller CPU 203 first sets the initial position coordinates of operation terminal 300 (wristband) as terminal position coordinates Now in step S9051.

  In this example, the initial position coordinates are the coordinates of the center of the operation terminal 300 when the wristband is installed at the installation location 811 as shown in FIG. 15, and the coordinate information is the operation terminal 300 or MFP1. It is held in advance by the main body.

  Next, in step S9053, the controller CPU 203 obtains an acceleration detection result by the three-dimensional acceleration sensor 303.

  Next, in step S9055, as described with reference to FIG. 8, the controller CPU 203 determines the moving distance of the operation terminal 300 from the acceleration a obtained from the acceleration in each direction and the elapsed time T (control cycle T). calculate.

  Next, in step S9057, the controller CPU 203 calculates a moving direction from the direction of the acceleration a.

  In step S9059, the controller CPU 203 calculates the latest current position coordinate Anew obtained by moving the current terminal position coordinate Anow by the movement distance in step S9055 in the movement direction in step S9057.

  In step S9061, the controller CPU 203 updates “Anow” to “Anew”.

  Next, in step S9063, the controller CPU 203 determines whether the coordinates in the area specified by the coordinate data group of the operation unit that is prohibited from operation in the control position information in FIG. If NO in step S9067, the flow advances to step S9067. If NO in step S9065, the flow advances to step S9065.

  In step S9065, the controller CPU 203 clears the count value of the operation counter Chnd and returns the process to step S90 in FIG. Thereby, the process proceeds from step S90 to step S50. The operation counter Chnd is a counter for determining whether or not the user wearing the operation terminal 300 is about to operate an operation unit that is prohibited from being operated.

  On the other hand, in step S9067, the controller CPU 203 determines whether or not the moving speed of the operation terminal 300 has exceeded the operation determination speed threshold (see FIG. 3) of the operation unit that is prohibited from being operated. Proceed to the process.

  On the other hand, if the controller CPU 203 determines that it does not exceed, the process directly returns to step S90. In this case, the process proceeds from step S90 to step S50.

  In step S9069, the controller CPU 203 increments the count value of the operation counter Chnd by 1, and advances the process to step S9071.

  In step S9071, the controller CPU 203 determines whether or not the count value of the operation counter Chnd has exceeded the gaze determination threshold value. If it is determined that the count value has exceeded, the process proceeds to step S9071. The gaze determination threshold value is a count value corresponding to the operation determination time threshold value (see FIG. 3) of the operation unit that is prohibited from being operated. Step S9071 is executed every control cycle T. Therefore, for example, a value obtained by dividing the operation determination time by the control cycle T is set as the gaze determination threshold value.

  If it is determined in step S9071 that the count value does not exceed the threshold value, the controller CPU 203 returns the process to step S90. In this case, the process proceeds from step S90 to step S50.

  In step S9073, the controller CPU 203 determines that the user's operation is directed to the operation unit where the operation is prohibited, and returns the process to step S90 (FIG. 5). As a result, the process proceeds from step S90 to step S100.

  In the example described with reference to FIGS. 15 and 16, the position of the operation terminal 300 is specified based on the detection output of the three-dimensional acceleration sensor 303. Note that an infrared sensor is attached to a specific part of the MFP 1 main body, and the controller CPU 203 detects how close the operation terminal 300 is to each infrared sensor based on the detection output of each infrared sensor, thereby three-dimensionally. The position of the operation terminal 300 specified based on the detection output of the acceleration sensor 303 may be corrected. As a result, the position of the operation terminal 300 can be specified more accurately, and as a result, it is possible to accurately detect what operation the user wearing the operation terminal 300 is performing. Note that the position of the operation terminal 300 described with reference to FIG. 4 may also be corrected by the detection output of the infrared sensor.

[Effects of the embodiment, etc.]
According to the present embodiment described above, the MFP 1 can detect a wide range of user operations without providing a large number of cameras and human sensors, and can warn the user of erroneous operations.

  Each embodiment disclosed this time must be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims. In addition, the invention described in the embodiment and each modification is intended to be carried out independently or in combination as much as possible.

  1 MFP, 1A toner replacement door, 1B front door, 10 image reading unit, 20 printer unit, 40 paper transport unit, 101 paper feed sensor, 110 timing sensor, 150K, 150Y, 150M, 150C imaging unit, 151K, 151Y, 151M , 151C ink cartridge, 200 controller, 203 controller CPU, 105, 202, 305 ROM, 204 terminal tracking control unit, 205 camera image processing unit, 206 image processing unit, 300 operation terminal, 304 operation terminal CPU, 302 gyro sensor, 303 Three-dimensional acceleration sensor, 307 camera, 309 panel display unit, 390 band, 811 installation location.

Claims (12)

An image forming system comprising an image forming apparatus and a terminal device capable of communicating with the image forming apparatus,
The image forming apparatus includes:
There are a plurality of operation units, and the operation procedure information when a state requiring a user operation on any one of the plurality of operation units occurs, and the operation by the user when the state occurs are prohibited. Holding the prohibited operation unit information, which is information of the operation unit.
The terminal device
A camera,
Control means for controlling the image forming apparatus,
Detecting means for detecting a position of the terminal device with respect to the image forming apparatus;
Warning means for outputting a warning to a user of the terminal device;
When the state occurs, the user's operation is monitored based on the photographing result of the camera, and when it is detected that the user has acted on the operation unit of the prohibited operation unit information, the warning unit And an output instruction means for outputting a warning to the image forming system.   The output instruction unit monitors a user's operation based on a result of photographing by the camera on the condition that the position detected by the detection unit is within a predetermined range from the position of the prohibited operation unit information. The image forming system according to claim 1.   The output instruction means, when it is detected that the line of sight of a person detected from the photographing result of the camera is continuously directed to the operation section of the prohibited operation section information, when the warning means is detected. The image forming system according to claim 1, wherein a warning is output to the image forming apparatus.   The output instructing unit causes the warning unit to output a warning when it is detected from a photographing result of the camera that the movement of a person is a movement toward the operation unit of the prohibited operation unit information. Item 4. The image forming system according to Item 1.   The output instruction means uses a speed measurement mechanism capable of measuring the speed of the user in the moving direction, and causes the warning means to output a warning when the speed of movement toward the operation unit is equal to or higher than a predetermined speed. The image forming system according to claim 4. Wherein the predetermined range is a predetermined range that is assumed the plurality of operation portions manually operable, an image forming system according to claim 2.   When the output instruction means detects that the user has acted on an operation unit other than the operation unit that requires user operation and the operation unit of the prohibited operation unit information among the plurality of operation units. The image forming system according to claim 1, wherein an operation unit different from the operation unit instructed by the procedure information is notified. The operation unit whose operation is prohibited by the prohibited operation unit information is held for each state,
The image forming system according to claim 3, wherein the predetermined time is set for each state. An infrared sensor for detecting a positional relationship between the image forming apparatus and the object;
The image forming system according to claim 1, wherein the detection unit corrects a position of the terminal device with respect to the image forming apparatus based on a detection output of the infrared sensor.   10. The image forming system according to claim 1, wherein the terminal device is movable apart from the image forming apparatus. 11. A control method executed by a computer for controlling an image forming system including an image forming apparatus and a terminal device capable of communicating with the image forming apparatus,
The image forming apparatus includes:
There are a plurality of operation units, and the operation procedure information when a state requiring a user operation on any one of the plurality of operation units occurs, and the operation by the user when the state occurs are prohibited. Holding the prohibited operation unit information, which is information of the operation unit.
The terminal device
A camera,
Control means for controlling the image forming apparatus,
Detecting a position of the terminal device with respect to the image forming apparatus;
Outputting a warning to a user of the terminal device;
When the state occurs, the user's operation is monitored based on the photographing result of the camera, and the warning is issued when it is detected that the user has acted on the operation unit of the prohibited operation unit information. And a step of outputting the image forming system. A control program executed by a computer for controlling an image forming system including an image forming apparatus and a terminal device capable of communicating with the image forming apparatus,
The image forming apparatus includes:
There are a plurality of operation units, and the operation procedure information when a state requiring a user operation on any one of the plurality of operation units occurs, and the operation by the user when the state occurs are prohibited. Holding the prohibited operation unit information, which is information of the operation unit.
The terminal device
A camera,
Control means for controlling the image forming apparatus,
The control program is stored in the computer.
Detecting a position of the terminal device with respect to the image forming apparatus;
Outputting a warning to a user of the terminal device;
When the state occurs, the user's operation is monitored based on the photographing result of the camera, and the warning is issued when it is detected that the user has acted on the operation unit of the prohibited operation unit information. And a control program for the image forming system.

Images