JP2019159939A - Electronic device and image forming apparatus - Google Patents

Abstract

To change the displayed details of a handling method displayed on a display unit according to the facial expression of a user to perform appropriate support according to the user's degree of understanding of the handling of an electronic device.SOLUTION: An image forming apparatus 10 comprises: a storage unit 28 that stores in advance a plurality of pieces of handling information different from each other in the amount of information indicating a method for handling the image forming apparatus 10, in association with the types of facial expression of a user; a facial expression detection unit 32 that, when the information on handling of the image forming apparatus 10 is displayed on a display unit 22, extracts a face image of the user from an image picked up by an imaging unit 26, and analyzes the extracted face image of the user to detect the type of the facial expression of the user; and a control unit 31 that reads out, from the storage unit 28, the handling information corresponding to the type of the facial expression detected by the facial expression detection unit 32, and displays the read-out handling information on the display unit 22.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an electronic device that displays a handling method on a display unit, and more particularly to a technique for changing display contents of the handling method according to a user's facial expression.

  An image forming apparatus, which is an example of an electronic device, causes an image reading unit to read an image of a document, causes the image forming unit to print the image on a recording sheet, and discharges the recording sheet. In such an image forming apparatus, the handling method of the image forming apparatus is displayed on the display unit so that the user can understand the handling method by viewing the screen of the display unit. For example, when a so-called jam occurs in which the recording paper is jammed in the middle of conveyance, a series of illustration images showing the operation procedure (handling method) for removing the recording paper is displayed on the display unit, so that the user can remove the jam. Support.

  On the other hand, in Patent Document 1, a user who uses a ticket vending machine is imaged, a user's face image is extracted, the facial expression of the user is confirmed based on the extracted face image, and a display screen of the ticket vending machine is displayed. There is shown a device for enlarging the fare display section displayed on the display screen of the ticket vending machine to make the fare display section easier to see when the expression of the face is difficult to see.

JP 2008-234401 A

  In the above image forming apparatus, the same handling method is displayed on the display unit regardless of whether or not the user is used to handling the image forming apparatus. For this reason, depending on the user, there is a problem that the content of the handling method displayed on the display unit is unnecessarily long or insufficient. In the above-mentioned Patent Document 1, although the fare display section displayed on the display screen of the ticket vending machine is enlarged according to the facial expression of the user, the contents displayed on the display screen are not changed, Cannot be resolved.

  The present invention has been made in view of the above circumstances, and by changing the display contents of the handling method displayed on the display unit according to the user's facial expression, according to the user's understanding of handling electronic devices. The purpose is to provide appropriate support.

  An electronic apparatus according to an aspect of the present invention shows a display unit, an imaging unit that captures an image of a user, and a method of handling the image forming apparatus, and a plurality of pieces of handling information having different amounts of information are expressed by facial expressions of the user When the display unit displays storage information stored in association with each type of the electronic device and the display unit, the face image of the user is extracted from the image captured by the imaging unit, A facial expression detection unit that analyzes the extracted facial image of the user and detects the facial expression type of the user, and stores the handling information corresponding to the facial expression type of the user detected by the facial expression detection unit A control unit that reads from the unit and displays the read handling information on the display unit.

  According to the present invention, by changing the display contents of the handling method displayed on the display unit according to the user's facial expression, it is possible to provide accurate support according to the degree of understanding of the user regarding the handling of the electronic device. .

1 is a cross-sectional view illustrating an image forming apparatus according to an embodiment of the present invention. 1 is a block diagram illustrating a main internal configuration of an image forming apparatus according to an exemplary embodiment. It is a figure which shows notionally the correspondence of detailed handling information, normal handling information, rough handling information, and the kind of user's facial expression. It is a flowchart which shows the process sequence for displaying the handling information corresponding to the kind of user's facial expression. It is a figure which shows an example of the display screen of a display part. It is a figure which shows an example of the display screen of a display part. It is a figure which shows an example of the display screen of a display part.

  Hereinafter, an electronic apparatus and an image forming apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a cross-sectional view showing an image forming apparatus according to an embodiment of the present invention. The image forming apparatus 10 is an example of an electronic apparatus according to the present invention. The image forming apparatus 10 according to the present embodiment is an MFP (multifunction machine) having a plurality of functions such as a copy function, a printer function, and a scanner function. The image forming apparatus 10 includes an image reading unit 11 and an image forming unit 12.

  The image reading unit 11 includes an image sensor that optically reads an image of a document. The analog output of the image sensor is converted into a digital signal, and image data indicating the image of the document is generated.

  The image forming unit 12 prints an image indicated by the image data or image data received from the outside on a recording sheet. The image forming unit 3M for magenta, the image forming unit 3C for cyan, and the image for yellow A forming unit 3Y and an image forming unit 3Bk for black are provided. In each of the image forming units 3M, 3C, 3Y, and 3Bk, the surface of the photosensitive drum 4 is uniformly charged, the surface of the photosensitive drum 4 is exposed, and an electrostatic latent image is formed on the surface of the photosensitive drum 4. The electrostatic latent image on the surface of the photosensitive drum 4 is developed into a toner image, and the toner image on the surface of the photosensitive drum 4 is primarily transferred to the intermediate transfer belt 5. As a result, a color toner image is formed on the intermediate transfer belt 5. This color toner image is secondarily transferred to the recording paper P conveyed through the conveyance path 8 from the paper supply unit 14 in the nip region N between the intermediate transfer belt 5 and the secondary transfer roller 6.

  Thereafter, the recording paper P is heated and pressurized by the fixing device 15, the toner image on the recording paper P is fixed by thermocompression bonding, and the recording paper P is discharged to the discharge tray 17 through the discharge roller 16.

  Next, a configuration related to the control of the image forming apparatus 10 will be described. FIG. 2 is a functional block diagram showing the main internal configuration of the image forming apparatus 10. As shown in FIG. 2, the image forming apparatus 10 includes a control unit 21, a display unit 22, a communication unit 23, an operation unit 24, a touch panel 25, an imaging unit 26, a sensor group 27, a storage unit 28, an image reading unit 11, and an image. The formation part 12 etc. are provided. These components can transmit / receive data or signals to / from each other through a bus.

  The display unit 22 includes a liquid crystal display (LCD), an organic EL (Organic Light-Emitting Diode) display, and the like.

  The communication unit 23 is a communication interface including a communication module, and performs data communication with an external terminal device (not shown).

  The operation unit 24 includes hard keys such as a numeric keypad, an enter key, and a start key.

  A touch panel 25 is overlaid on the screen of the display unit 22. The touch panel 25 is a so-called resistive film type or electrostatic capacity type touch panel, and detects the touch (touch) of the user's finger or the like with the touch position of the touch panel 25 together with the GUI on the screen of the display unit 22. Enter user instructions. Therefore, the touch panel 25 serves as an operation unit to which a user operation on the screen of the display unit 22 is input.

  The storage unit 28 includes a RAM, a large capacity HDD (Hard Disk Drive), and the like, and stores various data and programs.

  The control unit 21 includes a processor, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The processor is a CPU (Central Processing Unit), MPU, ASIC, or the like. The control unit 21 functions as the control unit 31 and the facial expression detection unit 32 when the control program stored in the ROM or the storage unit 28 is executed by the processor. In addition, each said structure of the control unit 21 may be comprised with a hardware circuit, respectively, not based on the operation | movement based on a control program.

  The control unit 31 controls overall operation of the image forming apparatus 10 and plays a role as a processing unit that executes various processes. The control unit 31 controls the display unit 22 to display various setting screens and information. The control unit 31 controls the communication operation of the communication unit 23.

  The imaging unit 26 is a camera including a lens and an imaging element, and is provided in the vicinity of the operation unit 24, and its field of view is directed obliquely upward in front of the image forming apparatus 10. The imaging unit 26 images a user who is operating the operation unit 24 and outputs image data indicating the captured image. For example, the imaging unit 26 captures a moving image or a still image, and outputs the image data to the facial expression detection unit 32 in a data format in which the facial expression detection unit 32 can detect a facial image and facial expression.

  The facial expression detection unit 32 performs a known facial expression recognition process on the image data captured by the imaging unit 26. For example, the facial expression detection unit 32 extracts the face image of the user from the captured image indicated by the image data by a face image extraction process using a known technique, analyzes the extracted face image, The nose and mouth are detected, the angle of the corner of the eye and the angle of the mouth edge are detected, and the facial expression is detected based on the angle of the corner of the eye and the angle of the mouth edge.

  In addition, the sensor group 27 includes a plurality of sensors that detect various defects of the image forming apparatus 10, and detection outputs of the sensors are input to the control unit 31. For example, each sensor detects a paper jam (jam), out of paper, out of toner, and the like as problems during execution of a job using a copy function or a printer function.

  When executing a job using the copy function or the printer function, the control unit 31 determines a defect such as jam, out of paper, or out of toner based on the detection output of each sensor of the sensor group 27, and interrupts the job when the defect occurs. .

  In the image forming apparatus 10 having such a configuration, when the user operates the operation unit 24 or the touch panel 25 to instruct the execution of a job using, for example, a copy function, the control unit 31 causes the image reading unit 11 to copy a document. The image is read, and the image of the original is printed on the recording paper by the image forming unit 12.

  Further, when executing the job as described above, the control unit 31 interrupts the job when determining the malfunction of the image forming apparatus 10 based on the detection output of each sensor of the sensor group 27. The control unit 31 reads handling information (handling method of the image forming apparatus 10) for solving this problem from the storage unit 28 and causes the display unit 22 to display the handling information. The user looks at the handling information displayed on the display unit 22 and performs an operation for solving the problem.

  This handling information is stored in the storage unit 28 in advance. For example, the handling information includes a plurality of illustration images constituting a GFI animation indicating a work procedure for solving the problem.

  In addition, a plurality of pieces of handling information for solving the defects are stored in the storage unit 28 for each of the plurality of types of defects. For example, detailed handling information suitable for users who do not understand the work procedure for solving the problem (handling information with a large amount of information), users who generally understand the work procedure for solving the problem Suitable normal handling information (handling information with a medium amount of information) and general handling information (handling information with a small amount of information) suitable for users who have a good understanding of the work procedure to solve the problem It is stored in the storage unit 28. For example, 15 illustration images (large amount of information) indicating the work procedure in detail as detailed handling information are stored in the storage unit 28, and 10 illustration images (intermediate information amount) normally indicating the work procedure as normal handling information. ) Is stored, and an image (small amount of information) indicating the work procedure only with characters is stored as concise handling information.

  Further, the plurality of handling information stored for each defect, that is, detailed handling information for solving the defect, normal handling information, and rough handling information, are associated with each type of facial expression of the user. Is stored in the storage unit 28. In the case of the example shown above, 15 illustration images showing the work procedure in detail are associated with an angry expression, 10 illustration images normally showing the work procedure are associated with no expression, A simple and simple image is associated with a laughing expression.

  FIG. 3 is a diagram conceptually showing the correspondence between the plurality of handling information and the type of facial expression of the user stored in the storage unit 28 for each of a plurality of types of defects. In FIG. 3, paper jam (jam), paper out, and toner out are illustrated as a plurality of types of defects.

  In the storage unit 28, the jam storage area JR stores 15 illustration images showing details of the work procedure for clearing the jam and an angry facial expression in association with each other. Normally, 10 illustration images and expressionless expressions are stored in association with each other, and a character-only image briefly indicating the work procedure and a laughing expression are stored in association with each other.

  Further, in the storage unit 28, 13 illustration images showing details of the work procedure for eliminating the paper break and an angry facial expression are stored in association with each other in the storage area KR for running out of paper. The eight illustrated images and the expressionless expression are stored in association with each other, and the character-only image and the laughing expression are simply stored in association with each other.

  Similarly, in the storage unit 28, 11 illustration images and an angry facial expression are associated and stored in the storage area TR for running out of toner, and 6 illustration images and no expression are associated and stored. A character-only image briefly indicating the work procedure and a laughing expression are stored in association with each other.

  That is, each piece of handling information stored in the storage unit 28 for each type of facial expression has a different amount of information for each type of defect.

  Here, for example, when the control unit 31 determines a defect and interrupts the job, first, the control unit 31 reads each image corresponding to the expressionless expression from the storage area of the storage unit 28 corresponding to the type of the defect, The respective images are displayed on the display unit 22. At this time, the control unit 31 starts imaging by the imaging unit 26.

  Then, the facial expression detection unit 32 extracts the user's face image based on the image data indicating the captured image acquired from the imaging unit 26, analyzes the extracted face image, and detects the facial expression of the user. When the facial expression of the user detected by the facial expression detection unit 32 is, for example, an angry facial expression or a laughing facial expression, the control unit 31 replaces each image corresponding to the expressionless expression displayed by default with anger. Each illustration image corresponding to the facial expression or each image corresponding to the laughing facial expression is displayed on the display unit 22.

  Next, processing for displaying handling information corresponding to a user's facial expression when a job is interrupted when a problem occurs in the image forming apparatus 10 will be described with reference to a flowchart shown in FIG.

  First, when executing a job using a copy function or a printer function (S101), the control unit 31 determines whether or not the job has ended (S102), and an image based on the detection output of each sensor of the sensor group 27. It is determined whether or not a failure has occurred in the forming apparatus 10 (S103).

  At this time, if the control unit 31 determines that a failure has occurred (S103 “No”) and determines that the job has ended (S102 “Yes”), the process of FIG. 4 ends.

  If the control unit 31 determines that a problem has occurred (S103 “Yes”) before determining that the job has ended (S102 “No”), the control unit 31 interrupts the job being executed at this time (S104). ). In S103, the control unit 31 determines the type of failure based on the detection output of each sensor of the sensor group 27. The control unit 31 obtains the images (here, 10 illustration images) as handling information corresponding to the determined type of defect and handling information corresponding to the expressionlessness. The images are sequentially displayed one by one on the display unit 22 as a default (S106: an example is shown in FIG. 5).

  For example, when the control unit 31 determines that the defect is a jam, the control unit 31 reads ten illustration images corresponding to the expressionless expression with reference to the jam storage area JR of the storage unit 28, and the control unit 31 reads the 10 illustration images. One illustration image is sequentially switched and displayed on the display unit 22 one by one. For this reason, if the user generally understands the work procedure for clearing the jam, the jam can be cleared by visually recognizing the explanation of each illustration image displayed on the display unit 22.

  In addition, the control unit 31 starts imaging by the imaging unit 26 from the time when the display of each illustration image corresponding to the expressionless expression is started (S107). When the imaging unit 26 is capturing a moving image, the control unit 31 continues imaging from the start of imaging, and when the imaging unit 26 is capturing a still image, the control unit 31 determines a predetermined cycle (for example, 1 second). ) To repeat taking a still image.

  Then, the control unit 31 determines whether or not the problem of the image forming apparatus 10 has been resolved based on the detection output of each sensor of the sensor group 27 (S108). The facial expression detection unit 32 acquires image data of a still image at the predetermined cycle based on the image captured by the image capturing unit 26 in the case of moving image capturing, or the image data from the image capturing unit 26 in the case of still image capturing. Is acquired based on the image data, and the extracted face image is analyzed to determine whether or not the facial expression of the user is an expressionless (S109). .

  At this time, the control unit 31 solves the problem of the image forming apparatus 10 while the facial expression detection unit 32 determines that the facial expression of the user is an expressionless (S108 “No”, S109 “Yes”). If it is determined that the job has been made (S108 “Yes”), the job is resumed (S110). Thereafter, the process returns to S101.

  Further, before the control unit 31 determines that the problem of the image forming apparatus 10 has been resolved (S108 “No”), the facial expression detection unit 32 determines that the facial expression of the user is not an expressionless (S109 “No”). The control unit 31 determines whether the facial expression detection unit 32 detects the facial expression of the user as an angry facial expression or a laughing facial expression (S111).

  For example, when the control unit 31 determines that the facial expression of the user is detected as an angry facial expression by the facial expression detection unit 32 (S111 “angry”), the control unit 31 uses the jamming of the storage unit 28. Referring to the storage area JR, 15 illustration images corresponding to the expression of anger are acquired, and the 15 illustration images shown in FIG. 6 are sequentially switched to the display unit 22 one by one. It is displayed (S112). Therefore, when the expression of the user's face is an angry expression, the display unit 22 displays 15 sheets showing the work procedure in more detail in place of the 10 illustration images that have been shown so far. An illustration image will be displayed. When the user has an angry facial expression in this way, it is assumed that the user cannot understand the work procedure even if he sees 10 illustration images, so it is assumed that the user is displeased and has an angry facial expression. Show 15 illustration images showing the work procedure to help them understand the work procedure.

  Then, the control unit 31 determines whether or not the problem of the image forming apparatus 10 has been resolved based on the detection output of each sensor of the sensor group 27 (S113), and keeps the above angry expression until the problem is resolved. The associated 15 illustration images are displayed on the display unit 22 (113 “No”, S112).

  When it is determined that the problem of the image forming apparatus 10 has been resolved (S113 “Yes”), the control unit 31 resumes the job that has been interrupted until then (S114). Thereafter, the process returns to S101.

  When the control unit 31 determines that the facial expression of the user is a laughing expression (S111 “laughing”), the control unit 31 refers to the jam storage area JR in the storage unit 28 to laugh. An image of only the character corresponding to the facial expression (an example is shown in FIG. 7) is read out, and the image of only the character is displayed on the display unit 22 by the control unit 31 (S115). Therefore, when the facial expression of the user is a laughing expression, instead of the ten illustration images that normally indicate the work procedure, an image of only the characters that briefly indicate the work procedure is displayed. In other words, if the facial expression of the user is a laughing expression, it is assumed that the user understands the work procedure for eliminating jamming, so simply display an image showing the work procedure and display the work procedure. Stop understanding. Thereby, wasteful display of the work procedure on the display unit 22 is prevented.

  Then, the control unit 31 determines whether or not the problem of the image forming apparatus 10 has been resolved based on the detection output of each sensor of the sensor group 27 (S116), and the control unit 31 performs the above process until the problem is resolved. An image of only characters associated with the laughing expression is displayed on the display unit 22 (116 “No”, S115).

  If the control unit 31 determines that the problem of the image forming apparatus 10 has been resolved (S116 “Yes”), the control unit 31 resumes the job that has been interrupted (S114). Thereafter, the process returns to S101.

As described above, in the present embodiment, the facial expression of the user is determined based on the image data indicating the captured image captured by the imaging unit 26, and the facial expression of the user is changed from an expressionless expression to an angry expression or a laughing expression. In the case of changing to, each illustration image corresponding to an angry expression or a laughing expression is displayed. Thus, by changing the display contents of the handling method displayed on the display unit 22 according to the user's facial expression, appropriate support according to the user's understanding of the work procedure for handling the image forming apparatus 10 is performed. For this reason, a user who does not understand the work procedure for solving the problem of the image forming apparatus 10 can show a detailed work procedure corresponding to the user's understanding accurately, or the image formation can be performed. The user who understands the work procedure for solving the malfunction of the apparatus 10 can prevent the display of the work procedure from being wasted by stopping the simple work procedure.
[Other embodiment 1]

  In the above-described embodiment, it is assumed that the captured image captured by the imaging unit 26 includes only the face image of one user, but the captured image includes multiple facial images. There is. As another embodiment, in such a case, the facial expression detection unit 32 detects each person's face image from the captured image, and is imaged with the largest area (maximum display size) among these face images. Is selected as a face image of the user who should show the work procedure by display on the display unit 22, and the selected face image is analyzed to change the facial expression to an expressionless expression, an angry expression, and a laughing expression. Detect which of the facial expressions. That is, in the other embodiment 1, in the above-described processing of S109 and S111, the facial image that the facial expression detection unit 32 detects the facial expression is the largest of the respective facial images of each person from the captured image. The face image is captured with an area (maximum display size).

Then, the control unit 31 causes the display unit 22 to display 10 illustration images associated with the expressionless expression while the facial expression indicated by the selected face image is expressionless (S106). , S109 “Yes”), (ii) When it is detected that the facial expression indicated by the selected face image is an angry facial expression, 15 illustration images are displayed on the display unit 22 (S111 “angry”). , S112), (iii) When it is detected that the facial expression indicated by the selected facial image is a laughing facial expression, an image of only characters is displayed on the display unit 22 (S111 "laughing", S115).
[Other embodiment 2]

  When a plurality of face images are included in the captured image, when the facial expression detection unit 32 detects a plurality of facial images from the captured image, the facial image is analyzed for each facial image to detect facial expressions. Then, (a) the control unit 31 causes the display unit 22 to display an illustration image corresponding to the expressionless expression while it is detected that all facial expressions are expressionless. (b) When the control unit 31 detects that the facial expression is an angry expression even for any one user, the control unit 31 displays an illustration image associated with the angry expression on the display unit. 22 is displayed. (c) Further, when it is detected that the facial expression is a laughing expression even for any one user, the control unit 31 displays an image associated with the laughing expression on the display unit 22. To display. That is, in the other embodiment 2, the facial expression detected by the facial expression detection unit 32 in the processing of S109 and S111 described above is the facial expression detected as described in the above (a) to (c).

  Further, the configuration and processing of the above-described embodiment described with reference to FIGS. 1 to 7 are merely examples of the present invention, and are not intended to limit the present invention to the configuration and processing. For example, in the above-described embodiment, an example of the electronic apparatus in the claims is described as the image forming apparatus 10, but the electronic apparatus is not limited to the image forming apparatus, and may be other electronic apparatuses such as medical devices and machine tools. It does not matter.

DESCRIPTION OF SYMBOLS 10 Image forming apparatus 11 Image reading part 12 Image forming part 21 Control unit 22 Display part 23 Communication part 24 Operation part 25 Touch panel 26 Imaging part 27 Sensor group 28 Storage part 31 Control part 32 Expression detection part

Claims (7)

A display unit;
An imaging unit for imaging a user;
A storage unit that indicates a handling method of the image forming apparatus, and stores a plurality of handling information having different amounts of information in association with each type of facial expression of the user,
When displaying the handling information of the electronic device on the display unit, the face image of the user is extracted from the image captured by the imaging unit, and the extracted face image of the user is analyzed to analyze the face of the user. A facial expression detection unit for detecting the type of facial expression;
An electronic apparatus comprising: a control unit that reads handling information corresponding to a type of facial expression of the user detected by the facial expression detection unit from the storage unit, and displays the read handling information on the display unit.   2. The electronic device according to claim 1, wherein each of the plurality of pieces of handling information is configured to include only characters that do not include an illustration image or includes a different number of illustration images. machine.   When the facial expression detection unit detects a plurality of facial images from the imaging unit, the facial expression detection unit detects the facial expression type of the facial image having the maximum display size from the facial images, and the detected facial facial expression The electronic device according to claim 1, wherein handling information corresponding to a type of the electronic device is read from the storage unit, and the read handling information is displayed on the display unit.   When the facial expression detection unit detects a plurality of facial images from the imaging unit, if at least one of the facial images shows a predetermined type of facial expression, the type of the facial expression The electronic device according to claim 1, wherein the handling information corresponding to is read from the storage unit, and the read handling information is displayed on the display unit.   5. The electronic device according to claim 1, wherein the facial expression detection unit detects a smiling facial expression, an angry facial expression, and no expression as the facial facial expression types. 6. The storage unit stores the plurality of pieces of handling information each having a different amount of information according to a predetermined type of failure that occurs in the electronic device,
The control unit, when a failure occurs in the electronic device, reads the plurality of handling information corresponding to the type of the failure from the storage unit, and causes the display unit to display the read handling information. The electronic device according to claim 5. An electronic device according to any one of claims 1 to 6,
An image forming apparatus further comprising: an image forming unit that forms an image on a recording medium.