JP2018078354A - Image processing system, mobile terminal device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To switch between a guide by voice and a guide by image according to operation which a user should perform.SOLUTION: A mobile terminal device 3 capable of short-range communication with an image processing device 2 is configured to, upon receiving from the image processing device 2 operation information on operation which a user should perform to cancel an error of the image processing device 2, guide the operation the user should perform by voice or image on the basis of the operation information. It switches between the guide by voice and the guide by image according to a type of operation the user should perform, or a holding state of the mobile terminal device by the user.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing system, a portable terminal device, and a program, and more particularly, to a technique for providing guidance for a user to cancel an error from a portable terminal device when an error occurs in the image processing system.

  2. Description of the Related Art Conventionally, in an image processing apparatus such as an MFP (Multifunction Peripherals) and a portable terminal device owned by a user, when an error such as a paper jam occurs in the image processing apparatus, the image processing apparatus is the portable terminal apparatus. A technique is known in which guidance information including voice guidance and image guidance is transmitted to the mobile terminal device so that an operation for canceling the error is guided to the user by both voice and image (for example, Patent Document 1).

Japanese Patent Laid-Open No. 2006-13987

  There are various types of errors that occur in the image processing apparatus. For example, there are errors that can be canceled by one-handed operation by the user, and errors that require the user to perform two-handed operation to cancel. If a two-handed operation is required to cancel the error, the user cannot perform the canceling operation while holding the mobile terminal device, and thus it is difficult to proceed with the canceling operation while viewing the image guidance. For this reason, when a two-handed operation is necessary, the user performs the releasing operation by relying only on the voice arrangement without looking at the image guidance. On the other hand, when the error can be canceled by one-handed operation, the user can perform the canceling operation with the other hand while holding the mobile terminal device with one hand. In this case, voice guidance is not particularly necessary because the user can accurately grasp the operation location by looking at the image guidance.

  However, since the above-described conventional technology always provides guidance by both sound and image, the mobile terminal device executes processing for image guidance even when the user is not viewing image guidance, Further, even when the user does not need voice guidance, the voice guidance processing is executed. Therefore, the above-described conventional technology has a problem in that it cannot switch between voice guidance and image guidance according to an operation to be performed by the user and consumes power of the mobile terminal device wastefully.

  The present invention has been made to solve the above-described conventional problems, and by switching between voice guidance and image guidance according to operations to be performed by the user, the power of the mobile terminal device An object of the present invention is to provide an image processing system, a portable terminal device, and a program that can reduce consumption.

  In order to achieve the above object, an invention according to claim 1 is an image processing system in which an image processing device and a mobile terminal device can perform short-range communication, and the image processing device includes the image processing device. An error detection means for detecting that an error has occurred, an operation information generation means for specifying an operation to be performed by the user in order to cancel the error detected by the abnormality detection means, and generating operation information relating to the operation; Operation information transmission means for transmitting operation information generated by the operation information generation means to the portable terminal device, wherein the portable terminal device receives operation information transmitted by the operation information transmission means Information receiving means and guidance that guides an operation to be performed by the user by voice or image based on the operation information received by the operation information receiving means The guidance means switches between voice guidance and image guidance according to the type of operation to be performed by the user or the holding state of the mobile terminal device by the user. It is.

  According to a second aspect of the present invention, in the image processing system according to the first aspect, the guide means performs a voice guidance when an operation to be performed by the user is an operation that needs to be performed with both hands, and is performed by the user. When the power operation is an operation that can be performed with one hand, the image guidance is performed.

  According to a third aspect of the present invention, in the image processing system according to the second aspect, in the image processing apparatus, the operation to be performed by the user specified by the operation information generating unit is an operation that needs to be performed with both hands. Or a discriminating unit that discriminates whether the operation can be performed with one hand, and the guidance unit switches between voice guidance and image guidance based on a discrimination result by the discrimination unit. This is a characteristic configuration.

  The invention according to claim 4 is the image processing system according to claim 2, wherein the portable terminal device is an operation that a user should perform with both hands by analyzing the operation information, Alternatively, the information processing apparatus further includes a determination unit that determines whether the operation can be performed with one hand, and the guide unit switches between voice guidance and image guidance based on a determination result by the determination unit. It is the structure to do.

  According to a fifth aspect of the present invention, in the image processing system according to the third or fourth aspect, the operation information generation unit specifies an operation for sequentially performing a plurality of steps one by one as an operation to be performed by the user. The discriminating means discriminates whether it is an operation that needs to be performed with both hands or an operation that can be performed with one hand for each step. According to the discrimination result, the guidance by voice and the guidance by image are switched at each step.

  According to a sixth aspect of the present invention, in the image processing system according to the fifth aspect, when the next step operation is a voice guidance operation, the guidance means uses an image before voice guidance. It is the structure characterized by performing guidance.

  According to a seventh aspect of the present invention, in the image processing system according to the fifth or sixth aspect, the operation information generating unit obtains operation information corresponding to the next step every time one step is advanced by a user operation. The operation information transmitting unit sequentially transmits the operation information for each step generated by the operation information generating unit to the portable terminal device, and the guide unit passes a predetermined time after performing the guidance by voice. When the operation information receiving means does not receive operation information corresponding to the next step, the user is prompted to switch from voice guidance to image guidance.

  According to an eighth aspect of the present invention, in the image processing system according to any one of the third to seventh aspects, the determination means uses both hands when the operation to be performed by the user is an operation of taking out the paper while operating the lever. In this configuration, it is determined that the operation needs to be performed.

  According to a ninth aspect of the present invention, in the image processing system according to any one of the third to eighth aspects, when the operation to be performed by the user is an operation of taking out paper while manually rotating a roller, both hands In this configuration, it is determined that the operation needs to be performed.

  According to a tenth aspect of the present invention, in the image processing system according to any one of the third to ninth aspects, when the operation to be performed by the user is an operation of supplying paper of a predetermined size or more, both hands. In this configuration, it is determined that the operation needs to be performed.

  According to an eleventh aspect of the present invention, in the image processing system according to any one of the first to tenth aspects, the mobile terminal device includes an imaging unit that captures an image, and a user's face from the image captured by the imaging unit. Image recognizing means for recognizing an image, and the guiding means performs guidance based on an image when the user's face image is recognized by the image recognizing means, and the user's face image is detected by the image recognizing means. In this configuration, voice guidance is performed when it is not recognized.

  The invention according to claim 12 is the image processing system according to any one of claims 1 to 11, wherein the mobile terminal device further includes an acceleration sensor that detects acceleration acting on the mobile terminal device, and the guidance The means is configured to perform guidance by image when acceleration is detected by the acceleration sensor, and to perform guidance by voice when acceleration is not detected by the acceleration sensor.

  According to a thirteenth aspect of the present invention, in the image processing system according to any one of the first to twelfth aspects, the image processing apparatus determines whether the operation performed by the user is a correct operation for canceling the error. An operation discriminating unit for discriminating, and when the guidance discriminating unit determines that the user's operation is an erroneous operation in the state of performing the voice guidance, the voice guidance is based on an image. In this configuration, the user is prompted to switch to guidance.

  According to a fourteenth aspect of the present invention, in the image processing system according to any one of the first to thirteenth aspects, the portable terminal device further includes an operation detecting unit that detects a user operation, and the guide unit is a voice. When the guidance is performed, the voice guidance is switched to the image guidance according to the user's operation detected by the operation detection means.

  According to a fifteenth aspect of the present invention, in the image processing system according to the fourteenth aspect, when the guidance unit performs guidance by voice, guidance by voice according to a user operation detected by the manipulation detection unit. Can be reproduced or paused.

  The invention according to claim 16 is a portable terminal device capable of short-range communication with an image processing device, wherein operation information related to an operation to be performed by a user in order to cancel an error of the image processing device, Operation information receiving means received from the image processing device, and guidance means for guiding the operation to be performed by the user by voice or image based on the operation information received by the operation information receiving means, the guidance means, According to the type of operation to be performed by the user or the holding state of the portable terminal device by the user, the voice guidance and the image guidance are switched.

  The invention according to claim 17 is the portable terminal device according to claim 16, wherein the guide means performs voice guidance when an operation to be performed by the user is an operation that needs to be performed by both hands, and is performed by the user. When the power operation is an operation that can be performed with one hand, the image guidance is performed.

  According to an eighteenth aspect of the present invention, in the portable terminal device according to the sixteenth or seventeenth aspect, when the operation to be performed by the user includes a plurality of steps, the guidance means uses voice guidance and an image for each step. It is the structure characterized by switching between guidance.

  According to a nineteenth aspect of the present invention, in the portable terminal device according to the eighteenth aspect, when the next step operation is a voice guidance operation, the guidance means uses an image before voice guidance. It is the structure characterized by performing guidance.

  According to a twentieth aspect of the present invention, in the portable terminal device according to the twelfth or nineteenth aspect, when the predetermined time has elapsed after the guidance unit performs voice guidance, the operation information reception unit performs the next step. When the operation information corresponding to is not received, the user is prompted to switch from voice guidance to image guidance.

  According to a twenty-first aspect of the present invention, in the portable terminal device according to any of the sixteenth to twentieth aspects, the guide means uses voice when an operation to be performed by the user is an operation of taking out paper while operating a lever. It is the structure characterized by performing guidance.

  According to a twenty-second aspect of the present invention, in the portable terminal device according to any of the sixteenth to twenty-first aspects, the guide means performs a voice operation when an operation to be performed by the user is an operation of taking out paper while manually rotating a roller. It is the structure characterized by performing guidance by.

  According to a twenty-third aspect of the present invention, in the portable terminal device according to any one of the sixteenth to twenty-second aspects, the guide means uses voice when the operation to be performed by the user is an operation of supplying paper of a predetermined size or more. It is the structure characterized by performing guidance.

  According to a twenty-fourth aspect of the present invention, in the mobile terminal device according to any one of the sixteenth to twenty-third aspects, an image recognition unit that captures an image and an image recognition that recognizes a user's face image from the image captured by the imaging unit. And when the user's face image is recognized by the image recognizing means, the guide means performs guidance by an image, and when the user's face image is not recognized by the image recognizing means. This configuration is characterized by performing guidance by voice.

  The invention according to claim 25 is the portable terminal device according to any one of claims 16 to 24, further comprising an acceleration sensor for detecting an acceleration acting on the portable terminal device, wherein the guiding means is the acceleration sensor. When the acceleration is detected by, guidance by image is performed, and when acceleration is not detected by the acceleration sensor, guidance by voice is performed.

  According to a twenty-sixth aspect of the present invention, in the portable terminal device according to any one of the sixteenth to twenty-fifth aspects, an operation in which a user's operation is incorrect in the image processing device when the guidance means performs a voice guidance. In the case where it is determined that it is, the user is prompted to switch from voice guidance to image guidance.

  The invention according to claim 27 is the portable terminal device according to any one of claims 16 to 26, further comprising operation detecting means for detecting a user operation, wherein the guide means performs voice guidance. The voice guidance is switched to the image guidance according to a user operation detected by the operation detection means.

  According to a twenty-eighth aspect of the present invention, in the portable terminal device according to the twenty-seventh aspect, when the guidance unit performs guidance by voice, guidance by voice according to a user operation detected by the manipulation detection unit. Can be reproduced or paused.

  The invention according to claim 29 is a program executed in a portable terminal device capable of performing short-range communication with an image processing device, in order to cancel an error of the image processing device in the portable terminal device. A first step of receiving operation information relating to an operation to be performed by the user from the image processing apparatus, and a first step of guiding the operation to be performed by the user by voice or image based on the operation information received in the first step. And the second step switches between voice guidance and image guidance according to the type of operation to be performed by the user or the holding state of the mobile terminal device by the user. It is the structure characterized by these.

  The invention according to claim 30 is the program according to claim 29, wherein the second step is performed by the user by performing voice guidance when an operation to be performed by the user is an operation that needs to be performed with both hands. When the power operation is an operation that can be performed with one hand, the image guidance is performed.

  The invention according to claim 31 is the program according to claim 29 or 30, wherein the mobile terminal device includes an image capturing unit that captures an image, and the program is captured by the image capturing unit on the mobile terminal device. A third step of recognizing the user's face image from the captured image, wherein the second step provides guidance by image when the user's face image is recognized in the third step, In the third step, voice guidance is performed when a user's face image is not recognized.

  The invention according to Claim 32 is the program according to any one of Claims 29 to 31, wherein the mobile terminal device includes an acceleration sensor that detects acceleration acting on the mobile terminal device, and the second step includes The image guidance is performed when acceleration is detected by the acceleration sensor, and the voice guidance is performed when acceleration is not detected by the acceleration sensor.

  According to the present invention, since the voice guidance and the image guidance are switched according to the operation to be performed by the user, the power consumption of the portable terminal device can be reduced.

It is a figure which shows the structural example of an image processing system. It is a block diagram which shows an example of the hardware constitutions of an image processing apparatus. It is a block diagram which shows an example of the hardware constitutions of a portable terminal device. It is a block diagram which shows the function structure of an image processing apparatus and a portable terminal device. It is a figure which shows an example of error discrimination information. It is a figure which shows an example of operation guidance registration information. It is a figure which shows an example of the image data registered into operation guidance registration information. It is a figure which shows an example of the audio | voice data registered into operation guidance registration information. It is a figure which shows the flow of the process which an image processing apparatus and a portable terminal device perform in cooperation with each other when an error occurs in the image processing apparatus. It is a flowchart which shows an example of the process sequence performed in an image processing apparatus. It is a flowchart which shows an example of the main process procedures performed in a portable terminal device. It is a flowchart which shows an example of the detailed process sequence of an operation guidance process.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In the embodiments described below, members that are common to each other are denoted by the same reference numerals, and redundant descriptions thereof are omitted.

  FIG. 1 is a diagram illustrating a configuration example of an image processing system 1 according to an embodiment of the present invention. The image processing system 1 includes an image processing apparatus 2 configured by an MFP or the like, and a mobile terminal apparatus 3 configured by a user, such as a smartphone, a tablet terminal, or a personal computer. The image processing device 2 and the mobile terminal device 3 are configured to be able to perform short-range communication with each other wirelessly.

  The image processing apparatus 2 includes a plurality of functions such as a scan function, a print function, and a copy function, and can execute a scan job, a print job, a copy job, and the like based on a user instruction. Such an image processing apparatus 2 may generate an error such as a paper jam inside. When an error occurs in the image processing device 2, the image processing device 2 causes the mobile terminal device 3 to perform operation guidance for error cancellation by performing short-range communication with the mobile terminal device 3 possessed by the user.

  An application for linking with the image processing apparatus 2 is installed in the mobile terminal device 3 in advance. The mobile terminal device 3 can perform near field communication with the image processing device 2 by activating the application. As the application is started in the state where an error has occurred in the image processing device 2, the mobile terminal device 3 starts near field communication with the image processing device 2, and an error is generated based on an instruction from the image processing device 2. The user is guided to the operation for canceling.

  FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image processing apparatus 2. The image processing apparatus 2 includes a control unit 10, an operation panel 11, a network interface 12, a storage unit 13, a near field communication interface 16, a scanner unit 20, and a printer unit 30.

  The control unit 10 includes a CPU 10a and a memory 10b, and controls the operation of each unit. When the image processing apparatus 2 is turned on, the CPU 10a reads the program 14 from the storage unit 13 and executes it, thereby causing the control unit 10 to function as various processing units. The memory 10b is for storing temporary data or the like when the CPU 10a executes processing based on the program 14.

  The operation panel 11 serves as a user interface when the user operates the image processing apparatus 2. The operation panel 11 includes a display unit 11a that displays various types of information to the user, and an operation unit 11b that receives an operation by the user. The display unit 11a is configured by a color liquid crystal display, for example. The operation unit 11b includes touch panel keys arranged on the screen of the display unit 11a and push button keys arranged around the screen of the display unit 11a. Such an operation panel 11 is fixed at a predetermined position on the front side of the main body of the image processing apparatus 2. Therefore, by standing on the front side of the image processing apparatus 2, the user can easily see the image displayed on the display unit 11a and can easily operate the operation unit 11b.

  The network interface 12 is used for the image processing apparatus 2 to communicate with a personal computer and other devices via a network such as a LAN (Local Area Network).

  The storage unit 13 is a non-volatile storage unit configured by, for example, a hard disk drive (HDD) or a solid state drive (SSD). In the storage unit 13, a program 14 that is executed by the CPU 10 a of the control unit 10 is stored in advance. In addition, error determination information 15 is stored in the storage unit 13 in advance. This error determination information 15 is referred to in order to specify a release operation to be performed by the user in order to release the error according to the type of error when an error such as a paper jam occurs in the image processing apparatus 2. Information.

  The near field communication interface 16 is an interface for performing near field communication with the mobile terminal device 3 possessed by the user. For example, the short-range communication interface 16 is an interface that performs wireless communication based on standards such as NFC (Near Field Communication) and Bluetooth (registered trademark).

  The scanner unit 20 reads a document set by a user and generates image data. The scanner unit 20 includes a document conveying unit 21 and an image reading unit 24. The document transport unit 21 automatically transports documents set by the user one by one along a predetermined transport path. The image reading unit 24 optically reads an image on the surface of the original when the original conveyed by the original conveying unit 21 passes a predetermined reading position.

  In such a scanner unit 20, there is a possibility that a paper jam occurs in the conveyance path when the document conveyance unit 21 performs automatic document conveyance. Therefore, the document conveying unit 21 includes a paper jam sensor 22 for detecting such a paper jam. In addition, the document transport unit 21 includes a cover (not shown) that can be opened and closed to open the transport path when such a paper jam occurs, and includes a cover open / close sensor 23 that detects the open / closed state of the cover. I have. The detection results of the paper jam sensor 22 and the cover opening / closing sensor 23 are output to the control unit 10.

  The printer unit 30 performs print output by forming an image on a print sheet based on the image data designated by the control unit 10. The printer unit 30 includes a paper feed unit 31, a paper transport unit 34, and an image forming unit 37.

  The paper feed unit 31 takes out and feeds sheets one by one from at least one tray (not shown) for stocking print sheets provided in the image processing apparatus 2. The paper feed unit 31 is provided with a paper empty sensor 32 for detecting paper empty in the tray. The paper feed unit 31 is provided with a tray open / close sensor 33 that detects the open / close state of the tray. The detection results by the paper empty sensor 32 and the tray opening / closing sensor 33 are output to the control unit 10. When the image processing apparatus 2 is provided with a plurality of trays, the paper empty sensor 32 and the tray opening / closing sensor 33 are provided for each tray.

  The paper transport unit 34 transports the paper fed by the paper feed unit 31 along a predetermined transport path. When the paper transport unit 34 is automatically transporting paper, a paper jam may occur in the transport path. Therefore, the paper transport unit 34 includes a paper jam sensor 35 for detecting such a paper jam. The paper transport unit 34 includes a door (not shown) that can be opened and closed to open the transport path when such a paper jam occurs, and a door open / close sensor 36 that detects the open / closed state of the door. I have. The detection results of the paper jam sensor 35 and the door opening / closing sensor 36 are output to the control unit 10. Note that the paper transport unit 34 may be provided with a plurality of paper jam sensors 35 or a plurality of door opening / closing sensors 36.

  The image forming unit 37 transfers the toner image to the surface of the sheet when the sheet conveyed by the sheet conveying unit 34 passes a predetermined position, and further, the sheet onto which the toner image is transferred passes another predetermined position. The toner image is fixed on the paper to form an image on the paper. The image forming unit 37 includes a toner remaining amount sensor 38. In addition, the image forming unit 37 includes an openable / closable door (not shown) that is opened when the toner is replaced, and includes a door open / close sensor 39 that detects an open / closed state of the door. The detection results of the remaining toner sensor 38 and the door opening / closing sensor 39 are output to the control unit 10.

  Next, FIG. 3 is a block diagram illustrating an example of a hardware configuration of the mobile terminal device 3. The mobile terminal device 3 includes a control unit 40, a display unit 41, an operation unit 42, a microphone 43, a speaker 44, an imaging unit 45, a storage unit 46, an acceleration sensor 49, and a short-range communication interface 50. It has.

  The control unit 40 includes a CPU 40a and a memory 40b, and controls the operation of each unit. The CPU 40a reads out and executes the program 47 stored in the storage unit 46, thereby starting an application for cooperating with the image processing apparatus 2, and causing the control unit 40 to function as various processing units described later. The memory 40b is for storing temporary data or the like when the CPU 40a executes processing based on the program 47.

  The display unit 41 is a display unit that displays various images to the user, and includes, for example, a color liquid crystal display. The operation unit 42 is for accepting an operation by a user, and includes, for example, touch panel keys arranged on the screen of the display unit 41, push button keys arranged around the screen of the display unit 41, and the like.

  The microphone 43 converts ambient sound into a sound signal and outputs the sound signal to the control unit 40. The speaker 44 outputs sound to the surroundings based on the sound signal input from the control unit 40. The imaging unit 45 is an imaging unit that is provided at a predetermined position around the display screen of the display unit 41, for example, and can capture an object in the normal direction with respect to the display screen of the display unit 41.

  The storage unit 46 is a nonvolatile storage unit configured by, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 46 stores the above-described program 47 in advance. Further, operation guide registration information 48 is stored in the storage unit 46 in advance. The operation guidance registration information 48 is information that is referred to by an application when an application that cooperates with the image processing apparatus 2 is activated in the control unit 40.

  The acceleration sensor 49 is a sensor that detects acceleration acting on the mobile terminal device 3. When the user has the mobile terminal device 3, the acceleration sensor 49 detects the acceleration accompanying the user's operation and outputs the acceleration to the control unit 40. On the other hand, when the user places the mobile terminal device 3 on a stationary object such as a floor surface, the acceleration sensor 49 does not detect acceleration, so the acceleration output to the control unit 40 is zero.

  The near field communication interface 50 is an interface for performing near field communication with the image processing apparatus 2. For example, the short-range communication interface 50 is an interface that performs wireless communication based on standards such as NFC (Near Field Communication) and Bluetooth (registered trademark), as with the image processing apparatus 2.

  In the image processing system 1 having the above configuration, when an error such as a paper jam occurs in the image processing apparatus 2 and the mobile terminal apparatus 3 guides an operation for canceling the error, the type of operation to be performed by the user, or The voice guidance and the image guidance are switched according to the holding state of the mobile terminal device 3 by the user. Hereinafter, the image processing system 1 will be described in more detail.

  FIG. 4 is a block diagram illustrating functional configurations of the image processing device 2 and the mobile terminal device 3. Note that FIG. 4 shows only a function for performing processing related to an error when an error occurs in the image processing apparatus 2, and does not show the function when the image processing apparatus 2 is operating normally. doing. 4 includes the paper jam sensor 22, the cover open / close sensor 23, the paper empty sensor 32, the tray open / close sensor 33, the paper jam sensor 35, the door open / close sensor 36, and the remaining toner sensor 38 shown in FIG. And a door opening / closing sensor 39.

  First, the control unit 10 of the image processing apparatus 2 functions as an error detection unit 61, an operation information generation unit 62, an operation determination unit 63, and an operation information transmission unit 64 when the CPU 10 a executes the program 14.

  The error detection unit 61 detects an abnormality as an error when any one of the plurality of sensors included in the sensor group 19 detects the abnormality. For example, when the paper jam sensors 22 and 35 detect a paper jam, when the paper empty sensor 32 detects paper empty, or when the toner remaining amount sensor 38 detects a toner empty pattern, the error detection unit 61 displays an image. It is detected that an error has occurred in the processing device 2. When detecting the occurrence of an error, the error detection unit 61 generates an alert indicating the occurrence of the error. Accordingly, the user can grasp that an error has occurred in the image processing apparatus 2. Further, when detecting the occurrence of an error, the error detection unit 61 causes the operation information generation unit 62 to function.

  When an error is detected by the error detection unit 61, the operation information generation unit 62 reads out and refers to the error determination information 15 from the storage unit 13 to identify an operation that should be performed by the user in order to cancel the error. , Generate operation information.

  FIG. 5 is a diagram illustrating an example of the error determination information 15. For example, the error determination information 15 is information in which the error type is classified into a plurality of types for each sensor that detects an error. Further, as shown in FIG. 5, the error determination information 15 defines an operation to be performed by the user in order to cancel the error for each sensor that detects the error. The operation to be performed by the user is composed of a plurality of steps, and the error can be appropriately canceled by sequentially performing the operations of each step. For example, three steps of the first step, the second step, and the third step are defined as the release operation when the paper jam sensor 22 provided in the document conveyance unit 21 of the scanner unit 20 detects a paper jam error. In the first step, the user performs an operation of opening the cover, in the second step, the user performs an operation of removing the jammed paper, and in the third step, the user performs an operation of closing the cover. Is defined. Further, for example, the first step, the second step, the third step, and the fourth step may be performed as a release operation when one paper empty sensor 32 provided in the paper feeding unit 31 of the printer unit 30 detects a paper empty error. In the first step, the user opens the door. In the second step, the user lifts the lever. In the third step, the user lifts the lever. In the fourth step, it is defined that the user performs an operation of closing the door. In the error determination information 15 shown in FIG. 5, operation information that is unique identification information is given for each operation step to be performed by the user.

  The operation information generation unit 62 refers to the error determination information 15 as described above, identifies the release operation corresponding to the sensor that has detected the error as an operation to be performed by the user, and sets each step of the identified operation. Operation information is generated by reading predetermined operation information. Then, the operation information generation unit 62 outputs the operation information of each step included in the release operation corresponding to the sensor that has detected the error to the operation information transmission unit 64.

  When an error is detected by the error detection unit 61, the operation information transmission unit 64 monitors whether communication with the mobile terminal device 3 is possible via the short-range communication interface 16. When the mobile terminal device 3 is in a communicable state, the operation information transmission unit 64 inquires of the mobile terminal device 3 whether or not an application that cooperates with the image processing device 2 is activated. If the application is activated in the mobile terminal device 3, the operation information transmission unit 64 receives an appropriate response to the inquiry from the application of the mobile terminal device 3. Thereby, the operation information transmission part 64 can grasp | ascertain that the application is starting in the portable terminal device 3 which can communicate. When the operation information transmission unit 64 detects that the application is activated in the mobile terminal device 3, the operation information transmission unit 64 transmits the operation information output from the operation information generation unit 62 to the mobile terminal device 3.

  On the other hand, when the operation information transmission unit 64 determines that the application is not activated in the communicable mobile terminal device 3, the operation information transmission unit 64 stands by until the application is activated in the mobile terminal device 3. However, the present invention is not limited to this, and the operation information transmission unit 64 may transmit an application activation command to the mobile terminal device 3. Thereby, an application is automatically started in the portable terminal device 3. The operation information transmission unit 64 transmits operation information output from the operation information generation unit 62 to the mobile terminal device 3 after confirming that the application is activated in the mobile terminal device 3.

  When transmitting the operation information to the mobile terminal device 3, the operation information transmitting unit 64 transmits the operation information of a plurality of steps included in the operation to be performed by the user to the mobile terminal device 3 one by one. For example, when an error is detected by the error detection unit 61, the operation information of the first step is transmitted to the mobile terminal device 3, and then the second step is performed when the operation by the user proceeds from the first step to the second step. Send operation information for. Thereby, the portable terminal device 3 can perform appropriate guidance according to the progress of the operation by the user.

  The operation determination unit 63 is a processing unit that monitors the outputs of a plurality of sensors included in the sensor group 19 and determines whether or not release operations performed by the user are performed in the correct order. For example, when the paper jam sensor 22 provided in the document conveying unit 21 of the scanner unit 20 detects an error, the first step of the operation to be performed by the user is an operation of opening the cover. Therefore, the operation determination unit 63 monitors the output of the cover opening / closing sensor 23 provided in the document conveyance unit 21 of the scanner unit 20 and determines whether or not the cover of the document conveyance unit 21 has been opened. When it is detected that the cover of the document transport unit 21 is opened, the operation determination unit 63 determines that the user's operation is performed in the correct order, and instructs the operation information transmission unit 64 to proceed with the step. To do. On the other hand, if the cover opening / closing sensor 23 does not detect the opening of the cover and another door opening / closing sensor 36, 39 detects the opening of the door, the operation determination unit 63 indicates that the operation by the user is an erroneous operation. to decide. In this case, the operation determination unit 63 outputs erroneous operation information to the operation information transmission unit 64. In addition, when an operation error is detected, the operation determination unit 63 may generate an alert indicating an operation error and may display an indication of an operation error on the display unit 11a of the operation panel 11.

  The operation information transmission unit 64 transmits the operation information corresponding to the next step to the portable terminal device 3 if the operation to be performed by the user has the next step when the operation determination unit 63 is instructed to proceed with the step. . On the other hand, if there is no operation information corresponding to the next step and it is determined that all steps have been completed, the operation information transmission unit 64 transmits an error release notification to the mobile terminal device 3. Thereby, the application activated in the portable terminal device 3 can end the cooperation processing with the image processing device 2.

  When the operation information transmission unit 64 acquires the erroneous operation information from the operation determination unit 63, the operation information transmission unit 64 transmits the erroneous operation information to the mobile terminal device 3. Thereby, the mobile terminal device 3 can notify the user that an erroneous operation has been performed.

  Next, when the mobile terminal device 3 starts the application that cooperates with the image processing device 2 in the control unit 40 by the CPU 40a executing the program 47, the control unit 40 causes the operation information reception unit 71 and the guidance method determination unit 72 to operate. And it functions as the operation guide part 73. The CPU 40a of the control unit 40 may execute the program 47 based on a start instruction by the user, or automatically execute the program 47 based on an application start command transmitted from the image processing apparatus 2. It may be executed. For example, when the occurrence of an error is detected in the image processing apparatus 2, an alert is issued from the image processing apparatus 2. Therefore, the user can grasp the occurrence of an error by an alert issued from the image processing apparatus 2 and can also start the application manually.

  The operation information receiving unit 71 communicates with the image processing apparatus 2 via the short-range communication interface 50. First, when the application is activated, the operation information receiving unit 71 notifies the image processing apparatus 2 of activation of the application. Thereby, the image processing apparatus 2 can grasp that the application is started in the mobile terminal device 3. Thereafter, the operation information receiving unit 71 receives operation information transmitted from the image processing apparatus 2. The operation information receiving unit 71 outputs the operation information to the guidance method determination unit 72 every time it receives operation information related to an operation to be performed by the user in order to cancel the error. The operation information receiving unit 71 also outputs erroneous operation information and error release notifications from the image processing device 2 to the guidance method determination unit 72 even when they are received.

  The guidance method discriminating unit 72 is a processing unit that discriminates whether to perform guidance by voice or guidance by image when guiding an operation to be performed by the user. When the operation information receiving unit 71 receives the operation information, the guidance method determining unit 72 first refers to the operation guide registration information 48 to determine the type of operation to be performed by the user.

  FIG. 6 is a diagram illustrating an example of the operation guidance registration information 48. As shown in FIG. 6, the operation guidance registration information 48 includes operation information 48 a for each step received from the image processing apparatus 2, image data 48 b for performing guidance by image, and audio data for performing guidance by voice. 48c and two-hand operation information 48d indicating whether or not a two-hand operation is necessary as an operation to be performed by the user, is registered information in association with each other. The image data 48b associated with the operation information 48a for each step is moving image data for guiding the operation corresponding to the step to the user by image display, and is data in which images are sequentially updated at a constant frame rate. . The audio data 48c associated with the operation information 48a for each step is audio data for guiding the operation corresponding to the step to the user by audio output, and is data in which time-series audio signals are recorded. The two-hand operation information 48d is information for determining whether or not the user needs to perform the operation in the step with both hands. When the user can perform the operation with one hand, “unnecessary” is recorded. ing.

  In the operation guidance registration information 48 shown in FIG. 6, when the operation information 48a is “C002”, “D002”, “D003”, “E002”, “E003”, “F002”, and “F003”, It is recorded that it is necessary.

  Here, referring to FIG. 5, the operation corresponding to the step of the operation information “C002” is an operation of supplying A3-size paper. On the other hand, in the case of an operation of supplying A4 size paper corresponding to the step “B002”, referring to FIG. 6, it is recorded that the two-hand operation is unnecessary. That is, the operation guidance registration information 48 shown in FIG. 6 is an operation that needs to be operated with both hands when the operation to be performed by the user is an operation for supplying paper of a predetermined size or more that needs to be held with both hands. There are registered.

  Referring to FIG. 5, the operation corresponding to the steps of the operation information “D002” and “D003” is an operation of pulling out a sheet with the lever lifted after the lever is lifted. That is, the operation guidance registration information 48 shown in FIG. 6 is an operation that needs to be performed with both hands when the operation to be performed by the user is an operation of taking out the paper with the other hand while operating the lever with one hand. Is registered.

  Referring to FIG. 5, the operation corresponding to the steps of the operation information “E002” and “E003” is an operation of removing the paper while rotating the roller. That is, the operation guidance registration information 48 shown in FIG. 6 is an operation that needs to be performed with both hands when the operation to be performed by the user is to manually rotate the roller with one hand and take out the paper with the other hand. It is registered.

  Still referring to FIG. 5, the operations corresponding to the steps of the operation information “F002” and “F003” are operations for exchanging toner. That is, in the operation guidance registration information 48 shown in FIG. 6, it is registered that the operation needs to be performed with both hands when it is an operation to replace the toner that needs to be held with both hands.

  FIG. 7 is a diagram illustrating an example of image data 48 b registered in the operation guidance registration information 48. 7 illustrates the image data 48b of each step when the paper empty sensor 32 (“paper empty sensor 1” shown in FIG. 5) detects an error. FIG. 7A shows image data 48b when the operation of the first step is guided to the user by an image. Since the operation to be performed by the user in the first step is an operation to open the tray 1, as shown in FIG. 7A, a moving image for opening the tray 1 is recorded in the image data 48b corresponding to the first step. ing. FIG. 7B shows image data 48b when the operation of the second step is guided to the user by an image. Since the operation to be performed by the user in the second step is an operation of supplying A4 size paper to the tray 1, as shown in FIG. 7B, the image data 48b corresponding to the second step includes A4 size paper. A movie to replenish is recorded. FIG. 7C shows image data 48b when the operation of the third step is guided to the user by an image. Since the operation to be performed by the user in the third step is an operation for closing the tray 1, as shown in FIG. 7C, a moving image for closing the tray 1 is recorded in the image data 48b corresponding to the third step. ing.

  FIG. 8 is a diagram illustrating an example of audio data 48 c registered in the operation guidance registration information 48. 8 exemplifies the audio data 48c at each step when the paper empty sensor 32 (“paper empty sensor 1” shown in FIG. 5) detects an error, as in FIG. FIG. 8A shows audio data 48c when the operation of the first step is guided to the user by an image. Since the operation to be performed by the user in the first step is an operation to open the tray 1, when the audio data 48c corresponding to the first step is reproduced as shown in FIG. The sound “Please open tray 1” is output. FIG. 8B shows audio data 48c when the operation of the second step is guided to the user by an image. Since the operation to be performed by the user in the second step is an operation of supplying A4-size paper to the tray 1, when the audio data 48c corresponding to the second step is reproduced as shown in FIG. The mobile terminal device 3 outputs a voice “please replenish A4 paper”. FIG. 8C shows audio data 48c when the operation of the third step is guided to the user by an image. Since the operation to be performed by the user in the third step is an operation to close the tray 1, when the audio data 48c corresponding to the third step is reproduced as shown in FIG. A sound “Please close tray 1” is output. However, such voice guidance is difficult for the user to accurately grasp the position of the tray 1 as compared with guidance by image.

  Thus, in the operation guidance registration information 48, image data 48b for performing guidance by image and voice data 48c for performing guidance by voice are registered in advance for each operation step to be performed by the user. . The image data 48b may be data including sound similar to the sound data 48c. However, if the audio is simultaneously output when displaying the moving image based on the image data 48b, the load for displaying the moving image and the load for reproducing the sound are simultaneously applied to the CPU 40a. The power consumption will be higher. In order to suppress this, it is preferable that the image data 48b is composed of data that does not include sound.

  The guidance method discriminating unit 72 discriminates whether or not the user needs to perform a two-hand operation in the step corresponding to the operation information received from the image processing device 2 by referring to the operation guidance registration information 48 as described above. To do. When it is necessary to perform a two-handed operation to cancel the error, the user cannot perform the operation with the portable terminal device 3 held in one hand. Therefore, even if the guidance based on the image based on the image data 48 b is displayed on the display unit 41 of the mobile terminal device 3, the user cannot see the guidance based on the image. Therefore, the guidance method discriminating unit 72 determines to perform voice guidance when it is necessary to perform a two-handed operation.

  On the other hand, when the operation can be performed with one hand, the user can perform the operation with the other hand while holding the mobile terminal device 3 in one hand. That is, when the operation can be performed with one hand, the user can proceed with the operation while viewing the guidance based on the image. Therefore, the guidance method discriminating unit 72 determines to perform guidance using an image when the user can operate with one hand in the step corresponding to the operation information received from the image processing apparatus 2.

  The guidance method determination unit 72 performs the determination process as described above every time operation information is received from the image processing apparatus 2. That is, the guidance method discriminating unit 72 determines whether guidance is performed using voice guidance or image guidance for each step included in the operation to be performed by the user.

  The guidance method determination unit 72 includes an image recognition unit 81, an operation detection unit 82, a voice recognition unit 83, and an acceleration determination unit 84. The guidance method determination unit 72 can switch between voice guidance and image guidance based on the detection results of the image recognition unit 81, the operation detection unit 82, the voice recognition unit 83, and the acceleration determination unit 84. Configured as follows.

  The image recognition unit 81 is a processing unit that drives the imaging unit 45 to acquire a captured image obtained by capturing the front side of the display unit 41 and recognizes whether the captured image includes a user's face image. For example, the image recognizing unit 81 drives the imaging unit 45 at a constant cycle and starts taking captured images at a constant cycle when an application is activated on the mobile terminal device 3 and a cooperative operation with the image processing device 2 is performed. get. Then, every time the captured image is acquired from the image capturing unit 45, the image recognizing unit 81 recognizes whether or not the captured image includes the user's face image. By performing such image recognition at regular intervals, it is possible to sequentially determine whether or not the user is viewing the display unit 41.

  When the image recognition unit 81 determines that the photographed image includes the user's face image, the guidance method determination unit 72 determines to perform guidance based on the image. That is, even if the operation to be performed by the user is an operation that needs to be performed with both hands, if there is a possibility that the user is viewing the display unit 41, it is determined to perform the guidance based on the image.

  On the other hand, when the image recognition unit 81 determines that the user's face image is not included in the captured image, the guidance method determination unit 72 determines to perform voice guidance. That is, even if the operation to be performed by the user is an operation that can be performed with one hand, if the user does not visually recognize the display unit 41, it is not necessary to perform guidance by image, but rather guidance by voice. Because it is preferable, it is decided to give guidance by voice. However, when the guidance method determined with reference to the operation guidance registration information 48 is guidance by image, guidance by image is performed without determining whether the photographed image includes the face image of the user. You may make it do. This is because guidance by image is easier for the user to understand than guidance by voice.

  The operation detection unit 82 detects a user operation on the operation unit 42. The user operation detected by the operation detection unit 82 includes an operation for switching from image guidance to voice guidance, an operation for switching voice guidance to image guidance, an operation to temporarily pause voice or image guidance, or voice or image. This includes operations for playing guidance. When the operation detection unit 82 detects these operations, the guidance method determination unit 72 performs processing according to the user's operation.

  The voice recognition unit 83 detects the user's voice operation on the portable terminal device 3 by recognizing the user's voice based on the voice signal output from the microphone. As described above, the user's voice operation detected by the voice recognition unit 83 is an operation for switching the image guidance to the voice guidance, the operation for switching the voice guidance to the image guidance, or the voice or image guidance is temporarily stopped. And an operation for reproducing guidance by voice or image. When the voice recognition unit 83 detects these operations, the guidance method determination unit 72 performs processing according to the user's voice operation.

  The acceleration determination unit 84 monitors the output of the acceleration sensor 49 and determines whether or not the acceleration sensor 49 is detecting acceleration. As described above, the acceleration sensor 49 detects the acceleration accompanying the user's operation when the user has the mobile terminal device 3, whereas detects the acceleration when the user does not have the mobile terminal device 3. do not do. Therefore, if the acceleration sensor 49 is detecting acceleration, it can be said that the user may be viewing the display unit 41 while holding the mobile terminal device 3. Therefore, the acceleration determination unit 84 determines whether there is a possibility that the user is viewing the display unit 41 based on the output from the acceleration sensor 49.

  When it is determined by the acceleration determination unit 84 that the acceleration is being detected, the guidance method determination unit 72 determines to perform image guidance. That is, even if the operation to be performed by the user is an operation that needs to be performed with both hands, if there is a possibility that the user is viewing the display unit 41, it is determined to perform the guidance based on the image.

  On the other hand, when the acceleration determination unit 84 determines that the acceleration is not detected, the guidance method determination unit 72 determines to perform guidance by voice. That is, even if the operation to be performed by the user is an operation that can be performed with one hand, if the user does not visually recognize the display unit 41, it is not necessary to perform guidance by image, but rather guidance by voice. Because it is preferable, it is decided to give guidance by voice. However, when the guidance method determined with reference to the operation guidance registration information 48 is guidance by image, guidance by image is performed without determining whether or not acceleration is detected. You may do it. This is because guidance by image is easier for the user to understand than guidance by voice.

  As described above, the guidance method determination unit 72 can switch between voice guidance and image guidance based on the determination results of the image recognition unit 81 and the acceleration determination unit 84. Therefore, the guidance method determination unit 72 can switch between voice guidance and image guidance according to the holding state of the mobile terminal device 3 by the user or the visual recognition state of the mobile terminal device 3 by the user. Further, in the guidance method discriminating unit 72, it is arbitrary which of the image recognition unit 81 and the acceleration judging unit 84 is preferentially adopted to switch the guidance method.

  The operation guidance unit 73 performs guidance processing for operations to be performed by the user based on the guidance method determined by the guidance method determination unit 72. The operation guide unit 73 includes a voice guide unit 74 and an image guide unit 75, and alternatively operates the voice guide unit 74 and the image guide unit 75 based on the determination result of the guide method determination unit 72.

  The voice guidance unit 74 is a processing unit that performs voice guidance. The voice guidance unit 74 reads the voice data 48c of the step corresponding to the operation information received from the image processing apparatus 2, and drives the speaker 44 based on the voice data 48c. The voice guidance is output from the speaker 44. That is, the voice guidance unit 74 causes the CPU 44a to perform voice reproduction processing based on the voice data 48c, thereby outputting guidance based on voice from the speaker 44. When the guidance method discriminating unit 72 gives an instruction to pause or reproduce the voice guidance, the voice guidance unit 74 pauses or plays the voice guidance based on the instruction from the guidance method discriminating unit 72. To do.

  When the voice guidance unit 74 provides guidance by voice, the operation guidance unit 73 may turn off the power supplied to the display unit 41. In this case, when the operation unit 42 detects the user's operation or when the microphone 43 detects the user's voice, the operation guide unit 73 causes the display unit 41 to convert the voice guidance into an image guidance. It is preferable to display a screen on which a switching operation, an operation for pausing voice guidance, an operation for reproducing voice guidance, and the like can be performed.

  The image guide unit 75 is a processing unit that performs image guidance. The image guide unit 75 reads the image data 48b of the step corresponding to the operation information received from the image processing apparatus 2, and displays a moving image on the display unit 41 based on the image data 48b. Display. That is, the image guiding unit 75 causes the display unit 41 to display guidance based on images by causing the CPU 40a to perform moving image reproduction processing based on the image data 48b. In addition, when the guidance method discriminating unit 72 gives an instruction to pause or reproduce the guidance based on the image, the image guiding unit 75 pauses or reproduces the guidance based on the image based on the instruction from the guidance method discriminating unit 72. To do. It should be noted that the operation guidance unit 73 may not drive the speaker 44 when the voice guidance unit 74 performs guidance by voice.

  In addition, the guidance method determination unit 72 operates the voice guidance unit 74 of the operation guidance unit 73 and operates the voice guidance unit 74 to receive erroneous operation information from the image processing apparatus 2 when performing guidance by voice. And instructing the user to switch from voice guidance to image guidance. Then, the operation guide unit 73 controls the voice guide unit 74 to output from the speaker 44 a voice prompting to switch the voice guidance to the image guidance. As a result, the user can easily understand the correct operation content because the user manually switches the voice guidance to the image guidance and visually recognizes the image guidance.

  In addition, the guidance method determination unit 72 provides voice guidance to the operation guidance unit 73 even when the operation guidance unit 73 determines that the user is troublesome in the operation. Instruct the user to switch to guidance by. For example, when a predetermined time has elapsed after the voice guidance unit 74 performs voice guidance, the operation information reception unit 71 receives the operation information corresponding to the next step. This is done by determining whether or not it is. When it is determined that the user is taking time for the operation, the operation guide unit 73 controls the voice guide unit 74 to output from the speaker 44 a sound prompting the user to switch from the voice guidance to the image guidance. As a result, the user switches the voice guidance to the image guidance manually and visually recognizes the image guidance. Therefore, the user can quickly grasp the correct operation content for canceling the error.

  In addition, when the guidance method determination unit 72 determines to perform guidance by voice based on the operation information received from the image processing apparatus 2, the guidance method determination unit 72 performs guidance by image before the operation guidance unit 73 performs guidance by voice. You may make it. It is difficult to specify the operation position etc. by voice guidance than the guidance by image, so it is easier for the user to specify the operation position etc. by showing the user the image guidance before giving the voice guidance. There are advantages.

  Further, the guidance method discriminating unit 72, at the timing when the user's face image is recognized by the image recognizing unit 81 or the acceleration is detected by the acceleration judging unit 84 during the voice guidance. The guidance method may be switched from voice guidance to image guidance.

  Next, FIG. 9 is a diagram illustrating a flow of a process performed when the image processing apparatus 2 and the mobile terminal apparatus 3 cooperate with each other when an error occurs in the image processing apparatus 2. As shown in FIG. 9, when the image processing apparatus 2 detects an error (process P1), it first notifies that an error has occurred (process P2). As a result, a user in the vicinity of the image processing apparatus 2 It can be understood that an error has occurred in the processing device 2. And the portable terminal device 3 starts an application based on the starting instruction | indication of the application by a user (process P3). However, the present invention is not limited to this, and the mobile terminal device 3 may automatically start an application based on a command from the image processing device 2.

  When the image processing apparatus 2 notifies the error, the image processing apparatus 2 specifies a canceling operation to be performed by the user in order to cancel the error (step S4). Then, the image processing device 2 generates operation information D10 of a step to be performed first by the user from among a plurality of steps included in the release operation (step S5), and transmits the operation information D10 to the mobile terminal device 3. (Process P6).

  When the portable terminal device 3 receives the operation information D10 transmitted from the image processing device 2 (process P7), the portable terminal device 3 determines a release operation guidance method for the user (process P8). At this time, the mobile terminal device 3 determines whether or not the operation to be performed by the user in the step corresponding to the operation information D10 received from the image processing device 2 is a two-handed operation. Either voice guidance or image guidance is determined according to whether a face image is included or whether acceleration is detected by the acceleration sensor 49. Then, the portable terminal device 3 guides an operation for canceling the error in either the voice guidance or the image guidance based on the determination result in the process P8 (process P9).

  Thereafter, when the image processing apparatus 2 detects that the operation step to be performed by the user has progressed based on the information obtained from the sensor group 19 (process P10), it generates operation information D11 corresponding to the next step, The operation information D11 is transmitted to the mobile terminal device 3 (process P11). Further, when receiving the operation information D11 of the next step transmitted from the image processing apparatus 2 (process P12), the mobile terminal device 3 determines the release operation guidance method for the user (process P13) in the same manner as described above. Based on the determination result, an operation for canceling the error is guided by either voice guidance or image guidance (process P14). Thereafter, the same processing is repeated until it is determined in the image processing apparatus 2 that the error has been canceled.

  Next, processing performed in the image processing apparatus 2 will be described. FIG. 10 is a flowchart illustrating an example of a processing procedure performed in the image processing apparatus 2. This process is started when an error is detected in the image processing apparatus 2. When starting this process, the image processing apparatus 2 first identifies the sensor that detected the error (step S10). If the sensor can be specified, the image processing apparatus 2 refers to the error determination information 15 (step S11), and specifies a release operation to be performed by the user (step S12). Then, the image processing device 2 identifies a step to be performed by the user from among a plurality of steps included in the identified release operation, generates operation information corresponding to the identified step, and transmits the operation information to the mobile terminal device 3 ( Steps S13 and S14).

  Thereafter, the image processing apparatus 2 monitors the output of each sensor included in the sensor group 19 and waits until it detects that an operation by the user has been performed (step S15). When it is detected that the operation by the user has been performed (YES in step S15), the image processing apparatus 2 determines whether or not the operation performed by the user is a correct operation based on the information obtained from the sensor group 19. It discriminate | determines (step S16). As a result, when the operation performed by the user is not a correct operation (NO in step S16), the image processing device 2 transmits erroneous operation information to the mobile terminal device 3 (step S17). On the other hand, when the operation performed by the user is a correct operation (YES in step S16), the image processing apparatus 2 determines whether or not the next step exists in the release operation to be performed by the user ( If there is a next step (YES in step S18), the process returns to step S13 and the above-described processing is repeated. On the other hand, when the next step does not exist (NO in step S18), the image processing apparatus 2 determines that the error has been released, and transmits an error release notification to the mobile terminal device 3 (step S19). Thus, the processing in the image processing apparatus 2 ends.

  Next, processing performed in the mobile terminal device 3 will be described. FIG. 11 is a flowchart illustrating an example of a main processing procedure performed in the mobile terminal device 3. This process is started when an application that cooperates with the image processing apparatus 2 is activated in the mobile terminal device 3. When starting this process, the mobile terminal device 3 first reads the operation guidance registration information 48 (step S20). And the portable terminal device 3 will be in the state which waits until operation information is received from the image processing apparatus 2 (step S21).

  When the operation information is received from the image processing device 2 (YES in step S21), the mobile terminal device 3 refers to the operation guide registration information 48 (step S22), and the operation to be performed by the user in this step is performed with both hands. Is determined whether or not the operation is necessary (step S23). As a result, when the operation can be performed with one hand (NO in step S23), the mobile terminal device 3 determines the guidance based on the image as the guidance method. On the other hand, if the operation needs to be performed with both hands (YES in step S23), the mobile terminal device 3 drives the imaging unit 45 (step S24) and acquires a captured image (step S25). Then, the mobile terminal device 3 determines whether or not the user's face image is included in the captured image (step S26). If the user's face image is not included (NO in step S26), the acceleration sensor 49 is further included. It is determined whether or not acceleration is being detected (step S27). And when the acceleration sensor 49 is not detecting acceleration (it is NO at step S27), the portable terminal device 3 determines the guidance by an audio | voice as a guidance method. Note that if the captured image includes a user's face image (YES in step S26) or if the acceleration sensor 49 is detecting acceleration (YES in step S27), the mobile terminal device 3 uses the guidance method. As a guide, an image guidance is determined.

  When the portable terminal device 3 determines the voice guidance, the portable terminal device 3 reads the voice data 48c corresponding to the current step (step S28), and starts the voice guidance by starting the reproduction process based on the voice data 48c (step S28). Step S29). And the portable terminal device 3 starts time measurement with starting voice guidance (step S30).

  Further, when the mobile terminal device 3 determines the guidance by the image, the mobile terminal device 3 reads the image data 48b corresponding to the current step (step S31), and starts the moving image reproduction process based on the image data 48b, thereby starting the guidance by the image. (Step S32).

  The portable terminal device 3 performs operation guidance processing when guidance by voice or image is started (step S33). FIG. 12 is a flowchart showing an example of a detailed processing procedure of the operation guidance process (step S33). When the mobile terminal device 3 starts the process shown in FIG. 12, it first determines whether or not voice guidance is being executed (step S40). Note that, here, it is determined as YES even if the next step is not performed after the voice guidance is performed. When the guidance by voice is performed (YES in step S40), the mobile terminal device 3 determines whether or not the time when the measurement is started in step S30 has passed the predetermined time (step S41), and the predetermined time is set. If it has elapsed (YES in step S41), a process of prompting the user to switch from voice guidance to image guidance is performed (step S42). Here, the predetermined time is, for example, about 3 to 5 minutes, and is set in advance enough to determine that the user is taking time for the release operation. On the other hand, when the guidance by voice is not performed (NO in step S40), or when the predetermined time has not elapsed (NO in step S41), the process of step S42 is skipped without being performed.

  Next, the mobile terminal device 3 determines whether or not erroneous operation information has been received from the image processing device 2 (step S43). When the erroneous operation information is received (YES in step S43), the mobile terminal device 3 notifies the user of the erroneous operation (step S44). This notification is performed, for example, by generating a warning sound from the speaker 44 of the mobile terminal device 3. Then, the mobile terminal device 3 determines whether or not voice guidance is performed in the same manner as in step S40 (step S45). If voice guidance is performed (YES in step S45), the mobile terminal device 3 prompts the user for voice. A process for prompting the user to switch from the guidance to the guidance based on the image is performed (step S46). Further, when the guidance by voice is not performed and the guidance by image is performed (NO in step S45), the mobile terminal device 3 reproduces the guidance by image again from the beginning (step S47). As a result, the user can perform a correct operation while confirming the guidance based on the image. If erroneous operation information has not been received (NO in step S43), steps S44 to S47 are skipped without being performed.

  Next, the portable terminal device 3 determines whether or not a user's manual or voice instruction is detected (step S48). When a user instruction is detected (YES in step S48), the mobile terminal device 3 determines whether or not the user instruction is a pause (step S49), and if it is a pause (YES in step S49). Then, the voice or image guidance pause processing is performed (step S50). If the user's instruction is not a pause (NO in step S49), it is determined whether the instruction is a voice or image guidance reproduction instruction (step S51), and if it is a reproduction instruction (YES in step S51). Then, reproduction processing of guidance by voice or image is performed (step S52). Further, if the user instruction is not a reproduction instruction (NO in step S51), it is determined whether or not it is a guidance method switching instruction (step S53), and if it is a guidance method switching instruction (YES in step S53). Then, a guidance method switching process is performed (step S54). In this guidance method switching processing, the mobile terminal device 3 performs processing for switching voice guidance to image guidance, or performs processing for switching image guidance to voice guidance. That is, the user can also switch the guidance method by his / her own operation. If the user's instruction is not a guidance method switching instruction (NO in step S53), the process skips without performing the process of step S54. In this case, processing according to other user instructions may be performed. Also, when the user's manual or voice instruction is not detected (NO in step S48), the above-described processes are skipped. This is the end of the operation guidance process (step S33).

  Returning to the flowchart of FIG. 11, the mobile terminal device 3 performs the operation guidance processing (step S33), and then determines whether or not an error cancellation notification is received from the image processing device 2 (step S34). If no error release notification has been received (NO in step S34), it is further determined whether or not operation information corresponding to the next step has been received (step S35). If the operation information corresponding to the next step has not been received (NO in step S35), the process by the mobile terminal device 3 returns to step S33, and the above-described process is repeated. When the operation information corresponding to the next step is received (YES in step S35), the process by the mobile terminal device 3 returns to step S22, and the above-described process is repeated. Further, when an error notification is received from the image processing device 2 (YES in step S34), the processing by the application running on the mobile terminal device 3 ends.

  As described above, when an error occurs in the image processing device 2, the mobile terminal device 3 according to the present embodiment receives operation information transmitted from the image processing device 2 and cancels the error based on the operation information. The operation to be performed by the user in order to do so is guided by voice or image. The mobile terminal device 3 is configured to switch between voice guidance and image guidance according to the type of operation to be performed by the user or the holding state of the mobile terminal device 3 by the user. Therefore, the mobile terminal device 3 according to the present embodiment does not guide the operation to be performed by the user with both sound and image as in the conventional case, and depends on the type of operation to be performed by the user, the holding state of the user, and the like. Thus, it is possible to selectively switch between voice guidance and image guidance, the load on the CPU 40a can be reduced at the same time, and the power consumption of the mobile terminal device 3 can be kept low. .

  Although one embodiment related to the present invention has been described above, the present invention is not limited to the contents described in the above embodiment, and various modifications can be applied.

  For example, the error of the image processing apparatus 2 described in the above embodiment is merely an example. Therefore, errors that occur in the image processing apparatus 2 may include errors other than those described above.

  Moreover, in the said embodiment, in the portable terminal device 3, when performing the process which discriminate | determines whether operation which a user should perform is operation which needs to be performed with both hands, or operation which can be performed with one hand Was illustrated. However, this determination process may be performed by the image processing apparatus 2. In other words, the image processing apparatus 2 determines whether the operation to be performed by the user to cancel the error is an operation that needs to be performed with both hands or an operation that can be performed with one hand. The determination result by the determination means may be transmitted to the mobile terminal device 3 together with the operation information. With such a configuration, it is necessary to perform processing for determining whether the operation to be performed by the user in the mobile terminal device 3 is an operation that needs to be performed with both hands or an operation that can be performed with one hand. There is an advantage that there is no.

  Moreover, in the said embodiment, the case where the portable terminal device 3 was comprised with the smart phone, tablet terminal, personal computer, etc. which are possessed exclusively by the user was illustrated. However, the mobile terminal device 3 does not necessarily have to be a device possessed exclusively by the user. For example, when the operation panel 11 of the image processing apparatus 2 is detachable from the apparatus main body of the image processing apparatus 2, the operation panel 11 removed from the apparatus main body of the image processing apparatus 2 is the mobile terminal device 3 described above. It may be functional.

DESCRIPTION OF SYMBOLS 1 Image processing system 2 Image processing apparatus 3 Portable terminal device 45 Imaging part (imaging means)
49 Acceleration sensor 61 Error detection unit (error detection means)
62 Operation information generation unit (operation information generation means)
63 Operation discrimination unit (operation discrimination means)
64 Operation information transmission unit (operation information transmission means)
71 Operation information receiving unit (operation information receiving means)
72 Guidance method discriminating section (discriminating means)
73 Operation Guide (Guide means)
81 Image recognition unit (image recognition means)

Claims (32)

An image processing system capable of short-range communication between an image processing device and a mobile terminal device,
The image processing apparatus includes:
Error detecting means for detecting that an error has occurred in the image processing apparatus;
An operation information generating means for specifying an operation to be performed by the user in order to cancel an error detected by the abnormality detecting means, and generating operation information related to the operation;
Operation information transmitting means for transmitting operation information generated by the operation information generating means to the portable terminal device;
With
The portable terminal device
Operation information receiving means for receiving operation information transmitted by the operation information transmitting means;
Based on the operation information received by the operation information receiving means, guidance means for guiding the operation to be performed by the user by voice or image;
With
The image processing system characterized in that the guidance means switches between voice guidance and image guidance according to the type of operation to be performed by the user or the holding state of the mobile terminal device by the user.   The guidance means performs guidance by voice when the operation to be performed by the user is an operation that needs to be performed with both hands, and guides by image when the operation to be performed by the user is an operation that can be performed with one hand. The image processing system according to claim 1, wherein the image processing system is performed. The image processing apparatus determines whether an operation to be performed by a user specified by the operation information generation unit is an operation that needs to be performed with both hands or an operation that can be performed with one hand Further comprising means,
The image processing system according to claim 2, wherein the guidance unit switches between voice guidance and image guidance based on a determination result by the determination unit. The mobile terminal device includes a determination unit that determines whether the operation to be performed by the user is an operation that needs to be performed with both hands or an operation that can be performed with one hand by analyzing the operation information. In addition,
The image processing system according to claim 2, wherein the guidance unit switches between voice guidance and image guidance based on a determination result by the determination unit. The operation information generation means specifies an operation for sequentially performing a plurality of steps one by one as an operation to be performed by the user,
The determination means determines, for each step, whether the operation needs to be performed with both hands or can be performed with one hand,
5. The image processing system according to claim 3, wherein the guidance unit switches between voice guidance and image guidance for each step based on a determination result for each step by the determination unit.   6. The image processing system according to claim 5, wherein when the next step operation is an operation for performing guidance by voice, the guidance means performs guidance by image before performing guidance by voice. The operation information generating means generates operation information corresponding to the next step each time one step is advanced by a user operation,
The operation information transmission means sequentially transmits operation information for each step generated by the operation information generation means to the portable terminal device,
The guidance means switches the voice guidance to the image guidance if the operation information receiving means has not received the operation information corresponding to the next step when a predetermined time has elapsed after the voice guidance. The image processing system according to claim 5, wherein the user is prompted to do so.   The determination unit determines that the operation to be performed by the user is an operation that needs to be performed with both hands when the operation is to take out the paper while operating the lever. The image processing system described.   9. The determination unit according to claim 3, wherein the determination unit determines that the operation to be performed by the user is an operation that needs to be performed with both hands when the operation is to take out the paper while manually rotating the roller. The image processing system described in 1.   10. The determination unit according to claim 3, wherein the determination unit determines that the operation to be performed by the user is an operation that needs to be performed with both hands when the operation is to replenish paper of a predetermined size or more. The image processing system described. The portable terminal device
Imaging means for taking an image;
Image recognition means for recognizing a user's face image from an image taken by the imaging means;
Further comprising
The guide means performs guidance by an image when a user's face image is recognized by the image recognition means, and performs voice guidance when the user's face image is not recognized by the image recognition means. The image processing system according to claim 1, wherein the image processing system is characterized in that: The portable terminal device
An acceleration sensor for detecting an acceleration acting on the portable terminal device;
Further comprising
12. The guidance device according to claim 1, wherein guidance is provided by an image when acceleration is detected by the acceleration sensor, and voice guidance is performed when acceleration is not detected by the acceleration sensor. An image processing system according to any one of the above. The image processing apparatus includes:
Operation discriminating means for discriminating whether or not the operation performed by the user is a correct operation for canceling the error;
Further comprising
The guidance means prompts the user to switch the voice guidance to the image guidance when the operation discrimination means determines that the user's operation is an incorrect operation in a state where the voice guidance is being performed. The image processing system according to claim 1, wherein the image processing system is characterized in that: The portable terminal device
Operation detection means for detecting user operations,
Further comprising
14. The guidance means switches voice guidance to image guidance according to a user operation detected by the operation detection means when performing voice guidance. The image processing system described in 1.   15. The guidance unit is capable of reproducing or pausing voice guidance according to a user operation detected by the operation detection unit when performing guidance by voice. The image processing system described in 1. A portable terminal device capable of short-range communication with an image processing device,
Operation information receiving means for receiving, from the image processing device, operation information related to an operation to be performed by a user in order to cancel an error of the image processing device;
Based on the operation information received by the operation information receiving means, guidance means for guiding the operation to be performed by the user by voice or image;
With
The portable terminal device characterized in that the guidance means switches between voice guidance and image guidance according to the type of operation to be performed by the user or the holding state of the portable terminal device by the user.   The guidance means performs guidance by voice when the operation to be performed by the user is an operation that needs to be performed with both hands, and guides by image when the operation to be performed by the user is an operation that can be performed with one hand. The mobile terminal device according to claim 16, wherein the mobile terminal device is performed.   The portable terminal device according to claim 16 or 17, wherein, when the operation to be performed by the user includes a plurality of steps, the guidance unit switches between voice guidance and image guidance for each step.   19. The portable terminal device according to claim 18, wherein when the next step operation is an audio guidance operation, the guidance means performs an image guidance before performing the voice guidance.   The guidance means switches the voice guidance to the image guidance if the operation information receiving means has not received the operation information corresponding to the next step when a predetermined time has elapsed after the voice guidance. 20. The portable terminal device according to claim 18, wherein the user is prompted to do so.   The portable terminal device according to any one of claims 16 to 20, wherein the guide means performs voice guidance when an operation to be performed by the user is an operation of taking out paper while operating a lever.   The portable terminal device according to any one of claims 16 to 21, wherein the guide means performs voice guidance when an operation to be performed by a user is an operation of taking out paper while manually rotating a roller.   The portable terminal device according to any one of claims 16 to 22, wherein the guidance unit performs voice guidance when an operation to be performed by the user is an operation of supplying paper of a predetermined size or more. Imaging means for taking an image;
Image recognition means for recognizing a user's face image from an image taken by the imaging means;
Further comprising
The guide means performs guidance by an image when a user's face image is recognized by the image recognition means, and performs voice guidance when the user's face image is not recognized by the image recognition means. The portable terminal device according to any one of claims 16 to 23, wherein An acceleration sensor for detecting an acceleration acting on the portable terminal device;
Further comprising
25. The guidance unit performs guidance by image when acceleration is detected by the acceleration sensor, and performs guidance by voice when acceleration is not detected by the acceleration sensor. The mobile terminal device according to any one of the above.   The guidance means prompts the user to switch the voice guidance to the image guidance when it is determined that the user's operation is an incorrect operation in the image processing apparatus during voice guidance. The mobile terminal device according to any one of claims 16 to 25, characterized in that: Operation detection means for detecting user operations,
Further comprising
27. The guidance means switches voice guidance to image guidance according to a user operation detected by the operation detection means when performing guidance by voice. The mobile terminal device according to 1.   28. The guidance means is capable of reproducing or pausing voice guidance according to a user operation detected by the operation detection means when performing guidance by voice. The mobile terminal device according to 1. A program executed in a portable terminal device capable of performing short-range communication with an image processing device, the portable terminal device,
A first step of receiving, from the image processing device, operation information relating to an operation to be performed by a user in order to cancel an error of the image processing device;
Based on the operation information received in the first step, a second step of guiding the operation to be performed by the user by voice or image;
And execute
The program in which the second step switches between voice guidance and image guidance according to the type of operation to be performed by the user or the holding state of the mobile terminal device by the user.   The second step performs guidance by voice when an operation to be performed by the user is an operation that needs to be performed with both hands, and uses an image when the operation to be performed by the user is an operation that can be performed with one hand. The program according to claim 29, wherein guidance is provided. The portable terminal device includes an imaging unit that captures an image,
The program is stored in the mobile terminal device.
A third step of recognizing a user's face image from an image photographed by the imaging means;
Is executed further,
The second step provides guidance by image when the user's face image is recognized in the third step, and voice guidance when the user's face image is not recognized in the third step. The program according to claim 29 or 30, wherein: The mobile terminal device includes an acceleration sensor that detects acceleration acting on the mobile terminal device,
30. The second step includes performing guidance by image when acceleration is detected by the acceleration sensor, and performing guidance by voice when acceleration is not detected by the acceleration sensor. 32. The program according to any one of thru 31.

Images