JP2020116778A - Image formation apparatus - Google Patents

Abstract

To provide an image formation apparatus which adopts a sight line detection technique and improves the convenience of a user.SOLUTION: A control unit analyzes data of an operator captured by a camera to execute a setting mode of setting the sight line of the operator to be detectable. The control unit starts capturing of the operator by the camera (S1). The control unit starts irradiation of the near-infrared ray light (S7), and starts sight line detection (S8). The control unit, when completing the sight line detection (YES in S10), displays the status of the sight line detection completion on a display unit so as to notify the operator of the completion of the sight line detection (S11). In other words, the control unit notifies the operator that the sight line detection of the operator becomes possible following execution of the setting mode. Since the operator can smoothly perform the transition to the sight line detection for the sight line input from the sight line detection as the preparation by notifying the operator that the sight line detection of the operator becomes possible, the convenience of the user is improved.SELECTED DRAWING: Figure 6

Description

The present invention relates to an image forming apparatus such as a printer, a copying machine, a facsimile, or a multi-function peripheral that employs the line-of-sight detection technology.

Recently, in order to improve the convenience of the operator (user), an image forming apparatus that employs a line-of-sight detection technique that detects the line of sight of the operator has been proposed (Patent Document 1). In the conventional device, control is performed to change the display displayed on the display device in response to detecting the line of sight of the operator directed to the display device (so-called line-of-sight input).

JP-A-2018-49528

However, although the line-of-sight detection technology has been conventionally used, it is not easy to use as an image forming apparatus. That is, in order to enable the line-of-sight input as described above, it is necessary to detect the line-of-sight of the operator. In the case of the device described in Patent Document 1, the operator's line of sight is detected by obtaining a reference point such as the operator's pupil or corneal reflection point. Therefore, since it is difficult to detect the line of sight until the measurement of the reference point is completed for each operator, the operator cannot correctly perform the input operation with the line of sight. However, in the past, since the operator had no way to know whether or not the reference point measurement was completed and the line-of-sight input was possible, the operator did not know whether to start the line-of-sight input and the convenience of the user was impaired. It was Therefore, it has been desired that the image forming apparatus adopts the line-of-sight detection technology and is more convenient for the operator, that is, more convenient for the user. However, such a thing is still proposed. Absent.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image forming apparatus that employs a line-of-sight detection technique and that improves user convenience.

An image forming apparatus according to the present invention is an image forming apparatus capable of forming an image on a recording material, and includes an operation unit including a user interface that allows an input operation by a user's line of sight, and an imaging unit that captures a face of the user. An image formation in accordance with a line-of-sight input operation on the user interface by the user's line-of-sight detected based on the setting of the setting mode in which the line-of-sight of the user whose face is imaged by the imaging unit is set to be detectable, And a control unit capable of executing an input mode for receiving an input regarding the input mode, wherein the control unit notifies the user that a line-of-sight input operation by the user's line of sight has become possible in the input mode. And

An image forming apparatus according to the present invention is an image forming apparatus that can form an image on a recording material, and includes an operation unit that includes a user interface that allows input by a user's line of sight, an imaging unit that images a user's face, A setting mode in which the line of sight of the user whose face is captured by the image capturing unit is set in a detectable state, and image formation in accordance with a line of sight input operation on the user interface by the line of sight of the user detected based on the setting of the setting mode A control unit capable of executing an input mode for receiving an input, wherein the control unit notifies the user that the setting to a state in which the user's line of sight can be detected in the setting mode has been completed. And

An image forming apparatus according to the present invention is an image forming apparatus that can form an image on a recording material, and includes an operation unit that includes a user interface that allows input by a user's line of sight, an imaging unit that images a user's face, A setting mode in which the line of sight of the user whose face is captured by the image capturing unit is set in a detectable state, and image formation in accordance with a line-of-sight input operation on the user interface by the line of sight of the user detected based on the setting of the setting mode And a control unit capable of executing an input mode for receiving an input, wherein the control unit notifies the user that the input mode has been changed from the setting mode.

According to the present invention, it is possible to improve user convenience in an image forming apparatus that employs the line-of-sight detection technology.

FIG. 1 is a schematic diagram showing an image forming apparatus of this embodiment. Schematic which shows an operation panel. The schematic diagram which shows the image|video of the operator's eyes imaged by the camera. The schematic diagram which shows an eyeball for demonstrating a visual line detection of an operator. The control block diagram for explaining a control part. The flowchart which shows the first half process of a line-of-sight input process. 7 is a flowchart showing a second half process subsequent to the first half process shown in FIG. 6. The figure explaining a line-of-sight correction. The figure explaining a gaze input operation. The figure which shows an example of a gaze input operation.

<Image forming device>
First, the image forming apparatus of this embodiment will be described with reference to FIG. The image forming apparatus 50 shown in FIG. 1 is a tandem type full color printer of an electrophotographic system. The image forming apparatus 50 prints a toner image on the recording material P ( It is formed on a sheet, a sheet material such as an OHP sheet).

The document reading device 601 has a document table on which a document can be placed, and can read the document placed on the document table, and a document feeding device 611 (Auto Document) that conveys the document to the reading device 610. Feeder). The reading device 610 is mainly provided with a first reading unit (not shown), a platen glass, and a flow reading glass. The document feeding device 611 can be opened and closed with respect to the document table glass of the reading device 610. The original placed on the platen glass is pressed by a pressure plate portion provided on the original conveying device 611 so as to face the platen glass, and an image is formed by a first reading unit which operates in response to the start of copying or scanning of the original. Read. On the other hand, the document feeding device 611 is mainly provided with a second reading unit and a document feeding unit (not shown). A document placed on a tray (not shown) is sent onto the flow reading glass of the reading device 610 by a document feeding unit that operates in response to the start of copying or scanning, and an image on the first side is printed by the first reading unit. The second reading unit reads the image on the second surface opposite to the first surface. The images read by the first reading unit and the second reading unit are used as image data for forming electrostatic latent images on the photosensitive drums a to d by the laser scanner 6 described later.

The image forming apparatus 50 includes an image forming unit 10 capable of forming yellow, magenta, cyan, and black images. The image forming unit 10 includes image forming units 1a, 1b, 1c, 1d arranged along the circumferential direction of the intermediate transfer belt 58. The image forming units 1a to 1d respectively include rotatable photosensitive drums a to d, a charger 60, and a developing device 59. The photosensitive drums a to d are charged by the corresponding charger 60, and then scanned and exposed by the laser light corresponding to the image data emitted from the laser scanner 6. As a result, electrostatic latent images corresponding to the image data are formed on the surfaces of the photosensitive drums a to d. Then, the electrostatic latent images formed on the photosensitive drums a to d are developed into a toner image by the developing device 59 using a developer. A primary transfer bias is applied to the toner images formed on the photosensitive drums a to d on the primary transfer rollers 2a to 2d arranged on the inner peripheral side of the intermediate transfer belt 58 so as to face the photosensitive drums a to d. As a result, primary transfer is performed on the intermediate transfer belt 58 at the primary transfer nip portion N1.

The endless belt-shaped intermediate transfer belt 58 is stretched by a secondary transfer inner roller 51 and stretching rollers 52 and 53. A secondary transfer outer roller 3a is arranged at a position opposed to the inner secondary transfer roller 51 with the intermediate transfer belt 58 interposed therebetween, and a secondary transfer is performed to secondarily transfer the toner image on the intermediate transfer belt 58 onto the recording material P. A transfer nip portion N2 is formed.

Below the image forming apparatus 50, the paper feed cassette 4 on which the recording material P is placed is arranged. The recording material P is supplied from the paper feed cassette 4 to the transport path 45 one by one by the feeding roller 8. The recording material P supplied from the paper feed cassette 4 is sent to the pair of registration rollers 9 through the transport path 45. The registration roller 9 temporarily receives the recording material P and corrects the skew of the recording material P, and then at the timing of forming the toner image on the intermediate transfer belt 58 in each of the image forming units 1a to 1d, the recording material P is formed. Is conveyed to the secondary transfer nip portion N2. Then, by applying a secondary transfer bias to the secondary transfer outer roller 3a, the toner image on the intermediate transfer belt 58 is secondarily transferred to the recording material P at the secondary transfer nip portion N2. After that, the recording material P is conveyed toward the fixing device 40 via the pre-fixing conveyance path 30. In the fixing device 40, as the recording material P is nipped and conveyed by the roller pair forming the fixing nip portion, the toner image is heated and pressed to be fixed on the recording material P.

When a toner image is formed on only one side of the recording material P, the recording material P that has passed through the fixing device 40 is discharged onto the paper discharge tray 12 by the pair of normal rotation paper discharge rollers 11. On the other hand, when the toner images are formed on both sides of the recording material P, the recording material P that has passed through the fixing device 40 is conveyed by the pair of normally rotating paper discharge rollers 11, but the rear end in the conveying direction reaches the reversal point 42. Then, switchback conveyance is performed toward the double-sided conveyance path 47. That is, when the trailing end of the recording material P in the conveying direction reaches the reversal point 42, the switching flapper 46 is switched and the rotation of the paper discharge roller pair 11 is changed from forward rotation to reverse rotation, and the recording material P is discharged from the paper discharge tray 12. It is sent to the double-sided transport path 47 without being discharged upward. The recording material P sent to the double-sided transport path 47 is returned to the transport path 45 by the pair of transport rollers 13, and the toner image is formed on the other surface through the same process as when forming the toner image on one side. .. Then, the recording material P having the toner images fixed on both surfaces is discharged onto the paper discharge tray 12 by the paper discharge roller pair 11 which rotates in the forward direction.

As shown in FIG. 1, the image forming apparatus 50 of the present embodiment includes a human sensor 450. The human sensor 450 is, for example, a temperature change detection type sensor that uses infrared rays and can detect a temperature change around the image forming apparatus 50. That is, when the operator approaches the image forming apparatus 50 and the ambient temperature changes by crossing the infrared rays emitted by the human sensor 450, the human sensor 450 can detect the whereabouts of the operator. In the case of the present embodiment, the human sensor 450 is provided on the operation panel so that the operator can detect the location of the image forming apparatus 50 when the operator is in front of the image forming apparatus 50, specifically, in a position facing the front of the operation panel 700. It is located at 700. In the present embodiment, the detectable range of the human sensor 450 is, for example, within the conical range with the human sensor 450 as the apex and about 500 mm in front of the human sensor 450 and a radius of about 300 mm. The human sensor 450 is not limited to a temperature change detection type thermal sensor using infrared rays, and any type of sensor may be used as long as it can detect the location of a person.

<Operation panel>
Further, the image forming apparatus 50 of this embodiment includes an operation panel 700. The operation panel 700 has an operator that can be operated by the operator, and is arranged on the front side of the apparatus main body 50a so that the operator can easily operate it. The operation panel 700 is provided in the apparatus main body 50a so that the angle can be adjusted vertically and horizontally by a tilt mechanism or a vari-angle mechanism. Therefore, the operator can adjust the orientation of the operation panel 700 so that his/her face and the front surface of the operation panel 700 face each other. FIG. 2 shows an example of an operation panel 700 as an operation unit. As shown in FIG. 2, the operation panel 700 includes a hardware unit 400 having physically provided hard keys 401, a display unit 410 capable of displaying various information and various screens including soft keys 403 described below, and an internal display unit 410. And a panel cover 710 that covers a provided substrate (not shown) and the like. The panel cover 710 is provided so that the hard part 400 and the display part 410 are exposed on the surface. In the case of the present embodiment, as will be described later, the hardware unit 400 and the display unit 410 of the operation panel 700 function as a user interface that enables an input operation by the user's line of sight.

<Hardware part>
The hard section 400 is provided with a hard key 401 which is a physical operator to which various functions such as a print key, a copy key, a numeric keypad, a stop key, a start key, and a power (on/off) switch are assigned. .. In the case of this embodiment, the hard key 401 is an operator with a backlight in which a light emitting LED 402 (Light Emitting Diode) is arranged on the back side of a transparent key or button, for example. That is, the LEDs 402 for light emission are arranged on each hard key 401 in order to individually light the hard keys 401. If the operator can distinguish each hard key 401 at a glance, a light emitting element such as a lamp may be arranged near each hard key 401 instead of the light emitting LED 402.

Various functions to be executed according to the operation by the operator may be previously assigned to each of the hard keys 401 described above. As an example, the print key is assigned a function of inputting various data relating to normal printing. The copy key is assigned a function of inputting various data relating to copy printing when the image forming apparatus 50 is used as a copying machine (referred to as a copy mode). Specifically, when the print key or the copy key is operated, as a function of inputting various data, a function of displaying an input screen for inputting various data on the display unit 410 is executed. An input function of inputting a numerical value such as the number of printed sheets is assigned to the ten-key pad. The stop key is assigned, for example, a function of forcibly stopping the image forming job being executed. The start key is assigned a function of instructing the start of execution of an image forming job including the reading operation of the original image in the copy mode. The power switch is assigned a function of turning on/off the power of the image forming apparatus 50.

<Display>
The display unit 410 appropriately displays various screens such as various data input according to the operation of the hard key 401, various information such as the operation status of the device (called status), and an input screen for inputting various data. You can Further, in the case of the present embodiment, the display unit 410 can display the soft key 403 as a virtual operator simulating the hard key 401 or the like. In FIG. 2, as the soft key 403, for example, a copy button (COPY) for copying, a scan button for starting document reading (SCAN), a button for fax transmission (FAX), and settings for displaying setting screens related to other functions Buttons (settings) and the like are shown as an example. When the soft key 403 is displayed, various functions pre-assigned to the soft key 403 are executed according to the touch operation of the display unit 410 on the soft key 403 by the operator. That is, the display unit 410 is a touch panel type display that can be touch-operated by the operator.

As will be described later in detail, in the case of the present embodiment, even if the operator does not actually touch and operate the hard key 401, the operator can turn the line of sight to the hard key 401 to perform the same hardware operation as when actually touching the hard key 401. The key 401 can be operated (so-called line-of-sight input). Further, when the soft key 403 is displayed on the display unit 410, the operator turns the line of sight to the desired soft key 403 without touching the soft key 403. It can be operated, that is, the line of sight can be input. In order to do so, in the image forming apparatus 50 of the present embodiment, the line-of-sight detection device 500 used to detect the line of sight of the operator is provided on the panel cover 710 which is a part of the operation panel 700. Since the line-of-sight detection device 500 is provided on the operation panel 700, the line-of-sight of the operator can be adjusted to a position that can detect the line-of-sight of the operator more accurately according to the angle adjustment of the operation panel 700.

<Gaze detection device>
As shown in FIG. 2, the visual line detection device 500 has a camera 510 and a near infrared light source 520. The camera 510 as an image pickup unit is, for example, an autofocus (AF) digital camera capable of photographing with infrared light, and has an image pickup element such as a CMOS sensor or a CCD sensor. The near-infrared light source 520 as an irradiation unit is, for example, an illuminating device that can irradiate invisible near-infrared light, and has a plurality of light-emitting elements such as near-infrared LEDs. The near-infrared light source 520 is arranged on the front surface of the image forming apparatus 50 so as to be able to emit near-infrared light toward an operator who operates the operation panel 700, and the camera 510 is operated by receiving the near-infrared light emitted from the near-infrared light source 520. The person's face (more specifically, the eyes) is arranged so that it can be photographed.

Further, the operation panel 700 is provided with a display lamp 420, a camera position display unit 421, a speaker 430, and a vibration generator 440. The camera position display unit 421 is, for example, a single-color (for example, red) LED or the like arranged at a predetermined position so that the operator can see the camera 510 by turning on the camera 510 during shooting. The display lamp 420 is, for example, a multi-color LED or the like for notifying the operator of the operation status of the device by light. The display lamp 420 is lit in green when the device is in a usable state, and is lit in red when the device is in a disabled state due to a paper jam or a failure. The speaker 430 is a sound generation unit capable of generating guidance for operation by a voice guide, a warning sound, and the like. It is preferable that the display lamp 420 and the speaker 430 are provided on the operation panel 700 whose position can be adjusted, because it is easy for the operator to be appropriately notified visually or by sound. Further, the operation panel 700 is provided with a vibration generator 440, and the vibration generator 440 can individually vibrate the hard keys 401.

<Gaze detection>
Here, the eye-gaze detection of the operator using the eye-gaze detection device 500 described above will be briefly described with reference to FIGS. 3 and 4. In order to detect the line of sight of the operator using the line-of-sight detection device 500, it is necessary to analyze the operator's data (specifically, eye data) captured by the camera 510. Therefore, in order to analyze this data, a control unit 300 (details of a line-of-sight detection unit 330, see FIG. 5) described later is provided. That is, in the present embodiment, the line-of-sight detection device 500 and the control unit 300 (specifically, the line-of-sight detection unit 330) described below constitute a line-of-sight detection unit capable of detecting the line of sight of the operator.

For the line-of-sight detection, a generally known line-of-sight detection method such as a corneal reflection method or a scleral reflection method may be adopted, and in the present embodiment, the corneal reflection method is used. The corneal reflection method specifies the reflection point center 761p of the reflection point 761 (Purkinje image) of the reflected light on the cornea Km surface of the operator's eye irradiated with near-infrared light and the pupil center 760p of the pupil 760 of the eyeball 750. Then, the line of sight of the operator is detected. That is, the direction of the operator's line of sight can be detected by capturing the position of the pupil center 760p with the position of the reflection point center 761p as the reference point. There are two methods of capturing the position of the pupil center 760p, the bright pupil method and the dark pupil method. The bright pupil method is a method of brightening the pupil to capture the position, and the dark pupil method is capturing the position by darkening the pupil. Is the way. In this embodiment, the dark pupil method is adopted.

To identify the position of the reflection point center 761p and the position of the pupil center 760p, first divide the data captured by the camera 510 into pixel units for the pupil 760 of the eye that has been irradiated with near-infrared light, the iris 762, and the reflection point 761. , Compare the brightness at each pixel. The brightness at each pixel is detected in the order of pupil <iris <reflection point. In accordance with this detection result, an image in which the boundaries of the pupil 760, the iris 762, and the reflection point 761 are captured is black-and-white binarized, and the center is obtained from the contours of the pupil 760 and the reflection point 761. The positions of 760p and the reflection point center 761p are specified.

When the positions of the reflection point center 761p and the pupil center 760p are specified, the line-of-sight vector Gb is obtained based on them. The line-of-sight vector Gb is obtained based on the corneal curvature radius Rk, the pupil radius Rd, and the distance Dkd from the corneal curvature center Kc to the pupil center 760p based on the registered eyeball model information (see FIG. 4). The corneal curvature center Kc and the pupil center 760p are obtained from two points of the pupil center 760p and the reflection point center 761p, the corneal curvature radius Rk, the pupil radius Rd, and the distance Dkd from the corneal curvature center Kc to the pupil center 760p. If the corneal curvature center Kc and the pupil center 760p are known, the line-of-sight vector Gb connecting them can be obtained. By obtaining the line-of-sight vector Gb, it is possible to capture the movement of the pupil 760 through the camera 510, and thus the line-of-sight of the operator can be detected. Here, the line of sight is the direction in which the operator is looking, and the point of sight is the viewpoint. In the present embodiment, by detecting this line of sight, it is detected which area on the operation panel 700 the operator is looking at. That is, in this embodiment, the viewpoint is detected by detecting the line of sight. In this way, by detecting the operator's viewpoint, it is possible to detect which area of the area on the operation panel 700 the operator is looking at. In the case of this embodiment, general average values may be used for the corneal curvature radius Rk, the pupil radius Rd, and the distance Dkd.

Note that it is preferable to perform line-of-sight correction (called calibration or the like) after specifying the positions of the pupil center 760p and the reflection point center 761p. The line-of-sight correction is performed to detect the line-of-sight of the operator with a smaller error.

<Control part>
As shown in FIG. 1, the image forming apparatus 50 of this embodiment has a control unit 300. The controller 300 will be described with reference to FIG. 5 while referring to FIG. 1 and FIG. In addition to the illustrated units, the control unit 300 includes, for example, the above-described image forming units 1a to 1d, motors for driving various rollers, a power supply for applying a primary transfer bias and a secondary transfer bias, and various devices such as the fixing device 40. Are connected and controlled by the control unit 300. However, since it is not the gist of the invention here, illustration and description thereof are omitted.

As illustrated in FIG. 5, the control unit 300 includes a memory 320 such as a CPU 310 (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The memory 320 serving as a storage unit stores various programs such as a line-of-sight input program (see FIGS. 6 and 7 described later) and an image forming job, and various data such as line-of-sight profile information described below and image data for displaying soft keys. Can be remembered. The CPU 310 can execute various programs stored in the memory 320. The memory 320 can also temporarily store the calculation processing result and the like accompanying the execution of various programs.

In the case of the present embodiment, the CPU 310 can control various operations of the image forming apparatus 50 relating to the operator's line-of-sight input by executing the line-of-sight input program. The line-of-sight input program is not limited to the form of a software program, and may be implemented in the form of a micro program processed by a DSP (digital signal processor), for example. That is, the CPU 310 may be configured to execute a control program such as an image forming job to perform various controls such as an image forming operation, but the present invention is not limited to this, and it is prepared specifically for executing the line-of-sight input program. You may use the thing.

A hard key 401, a light emitting LED 402, a display unit 410, a display lamp 420, a speaker 430, a vibration generator 440, a human sensor 450, and a line-of-sight detection device 500 are connected to the CPU 310 via, for example, a data bus and an address bus. Has been done. The CPU 310 can acquire various data input by, for example, an operation of the hard key 401 by the operator or a touch operation on the display unit 410, and store the data in the memory 320.

In the case of the present embodiment, the CPU 310 reads the line-of-sight input program (see FIGS. 6 and 7) from the memory 320 and executes the program when the operator's location is detected by the human sensor 450 described above. The control unit 300 can function as the line-of-sight detection unit 330, the notification control unit 340, the display control unit 341, the function execution unit 350, the information creation unit 360, and the like when executing the line-of-sight input program.

The line-of-sight detection unit 330 acquires data captured by the camera 510 from the line-of-sight detection device 500 and analyzes the acquired data. Thereby, the line-of-sight of the operator can be detected using the line-of-sight detection device 500. In addition, the line-of-sight detection unit 330 can perform line-of-sight correction. As a result, it is possible to detect the line of sight more accurately. Thus, in the present embodiment, the line-of-sight detection unit 330 realizes the line-of-sight detection of the operator using the line-of-sight detection device 500. The notification control unit 340 executes the control of notifying the operator that the line of sight of the operator has been detected by the display lamp 420, the speaker 430, the vibration generator 440, the light emitting LED 402, and the like as the notification means. The display control unit 341 can control the display unit 410 as a notification unit (display unit). The display control unit 341 causes the display unit 410 to display the above-described soft keys 403, status, and various screens. In the case of the present embodiment, the display control unit 341 notifies that the operator's line of sight has been detected as a status, or changes the display of the soft key 403 to which the line of sight is directed based on the detected line of sight of the operator, etc. You can then inform.

The function execution unit 350 executes various functions pre-assigned to these operators by the operation of the hard key 401 or the soft key 403 by the operator. The operation performed by the operator on the hard keys 401 and the soft keys 403 is not limited to a direct operation, but may be an indirect operation by the line-of-sight input based on the line-of-sight detection of the operator. As will be described later, the information creating unit 360 as an information creating unit creates line-of-sight profile information that is information about the line-of-sight of each operator (each user), and stores the created line-of-sight profile information in the memory 320.

<Gaze input program>
Next, the line-of-sight input program of this embodiment will be described with reference to FIGS. 3 to 5 and FIGS. 6 and 7. The line-of-sight input processing shown here is started by the control unit 300 when the presence sensor 450 detects the location of the operator, and ends when the presence sensor 450 no longer detects the location of the operator. For convenience of illustration, the first half processing of the line-of-sight input processing is shown in FIG. 6, and the second half processing subsequent to the first half processing is shown in FIG. 7.

In the present embodiment, first, by analyzing the data of the operator, which is the imaged result captured by the camera 510, the line-of-sight detection unit 330 for detecting the line of sight of the operator is set, and then the imaged result is displayed. Based on this, a setting mode for detecting the line of sight of the operator is executed. As shown in FIG. 6, the control unit 300 starts capturing an image of the operator with the camera 510 (S1). In response to the start of shooting, the control unit 300 displays "start shooting" on the display unit 410 (S2). The control unit 300 also turns on the camera position display unit 421 (S3), and causes the display unit 410 to display “Please look at the red portion on the panel” as the status (S4). As a result, by making the operator see the camera 510, the camera 510 can appropriately capture the operator's face. Then, the control unit 300 starts face detection based on the data captured by the camera 510 (S5). The control unit 300 causes the device to wait until the face detection is completed (NO in S6), and when the face detection is completed, the process proceeds to the next step S7 (YES in S6). In this face detection, different eye positions are specified for each operator in order to properly detect the line of sight of the operator. The operator adjusts the position of the operation panel 700 so that the face faces the front of the operation panel 700, so that the camera 510 can photograph the operator's face at a position suitable for face detection.

The face detection described above may use a known face recognition function adopted in a general digital camera. That is, the horizontal band pass filter and the vertical band pass filter are applied to the data captured by the camera 510 to detect the edge component from the data. Then, by extracting the candidate group of eyes, nose, mouth and the like by pattern matching between the face pattern based on the detected edge component and the various face patterns stored in the memory 320, the data is a face. Is recognized.

When the face detection is completed, the control unit 300 starts irradiating the near infrared light from the near infrared light source 520 (S7), and starts the line-of-sight detection using the corneal reflection method of the dark pupil method described above (S8). At that time, the control unit 300 keeps the camera position display unit 421 continuously lit and displays the status on the display unit 410 such as "Start line-of-sight detection. Look at the red part on the panel." (S9). In this way, by causing the operator to look at the camera 510 in the state where the light is radiated and taking a picture of the operator, the image of the reflected light on the corneal surface by the near-infrared light emitted by the near-infrared light source 520 (Purkinje image) Is displayed clearly in the eyes of the operator.

Then, the control unit 300 makes the apparatus stand by until the line-of-sight detection is completed (NO in S10), and when the line-of-sight detection is completed, the process proceeds to the next step S11 (YES in S10). In the case of the present embodiment, the control unit 300 informs the operator that the sight line detection has been completed and prepares for proceeding to the next sight line correction process by displaying “The sight line detection has been completed. It will be done." is displayed (S11).

In the present embodiment, by displaying the above status on the display unit 410, the operator is notified that the line-of-sight of the operator can be detected. Is not limited to this. For example, the display lamp 420 may be used. In that case, the display lamp 420 may be "blinked" when the setting mode is started, and the display lamp 420 may be "lighted" when the setting mode is ended and the input operation by the line of sight of the operator becomes possible. Alternatively, the operator may be notified by a voice from the speaker 430 that the input operation by the operator's line of sight has become possible, or the vibration generator 440 provided on the operation panel 700 may be vibrated continuously or simply. The operator may be informed.

In the present embodiment, after completion of the sight line detection, the information acquisition mode for continuously performing the sight line correction (calibration) and creating the sight line profile information can be executed. That is, as shown in FIG. 7, the control unit 300 starts line-of-sight correction (calibration) after completion of line-of-sight detection (S12). In the present embodiment, as an example of the line-of-sight correction, a method of allowing the operator to stay at a plurality of points is used. That is, the operator sequentially guides the operator to see three or more locations on the operation panel 700, and stops the line of sight so as to see each location for a predetermined time or longer. And the positional relationship of the reflection point center 761p (see FIG. 4) is obtained. Then, the visual axis detection result (see S10) is corrected based on the obtained positional relationship between the pupil center 760p and the reflection point center 761p and the positional relationship between the points on the operation panel 700. The line-of-sight detection result thus line-of-sight corrected is stored in the memory 320 as line-of-sight profile information for each operator.

Here, the conditions required for the line-of-sight correction will be described with reference to FIG. 5 and FIG. The operation panel 700 is preliminarily provided with addresses divided into a mesh shape so that the line-of-sight detection unit 330 can identify each location on the operation panel 700. In the present embodiment, as shown in FIG. 8, the row direction is divided into 10 rows indicated by “AA” and “KK”, and the column direction is divided into 13 columns indicated by “01” to “13”, and “10 rows× 13 columns=130 locations”. Absolute coordinates in the X direction, Y direction, and Z direction are attached to this address as absolute coordinates in the three-dimensional space, and are stored in the memory 320 as basic data for calculation by the line-of-sight detection unit 330.

In addition, markers 751, 752, and 753 are provided at three locations among the four corners of the operation panel 700 as target portions that are visually recognized by the operator for line-of-sight correction. Here, the marker 751 is provided in the upper left corner of the operation panel 700, the marker 752 is provided in the upper right corner of the operation panel 700, and the marker 753 is provided in the lower right corner of the operation panel 700. These markers 751 to 753 are provided so that no other marker is located on a straight line connecting any two markers. The markers may be provided on at least two locations on the operation panel 700. However, it is preferable that at least two of the markers are provided at the corners of the operation panel 700.

The target portion is formed in a quadrangular shape of, for example, 5 mm×5 mm so that the operator can easily recognize it, and is painted in a conspicuous color (for example, a color different from the color of the panel cover 710 such as yellow or blue) like the markers 751 to 753. It may have been done. Alternatively, it may be formed so as to emit light by using an LED or the like. Further, it may be formed by attaching a seal member to the panel cover 710. Note that the size of each target portion is not limited to the above-described size as long as it is a size that is easily visible to the operator and that the line of sight can be corrected. The markers 751 to 753 as the target portions are provided with absolute coordinates in the X direction, Y direction, and Z direction as absolute coordinates in the three-dimensional space. For example, the center position coordinates of the marker 751 (address AA01) are “X=AA01x, Y=AA01y, Z=AA01z”. The center position coordinates of the marker 752 (address AA13) are “X=AA13x, Y=AA13y, Z=AA13z”. The center position coordinates of the marker 753 (address KK13) are “X=KK13x, Y=KK13y, Z=KK13z”. The center position coordinates of these markers 751 to 753 may change depending on the angle of the operation panel 700.

Returning to the explanation of FIG. 7, the control unit 300 informs the operator that the eye gaze correction will be performed. The display unit 410 says “Start eye gaze correction. Look at the marker on the upper left of the operation unit.” The status is displayed (S13). Then, the control unit 300 determines whether or not the information collection necessary for the line-of-sight correction has been completed by the stationary view of the marker 751 based on the data of the operator who fixedly views the marker 751 captured by the camera 510 ( S14). The control unit 300 makes the operator stop the marker 751 and causes the camera 510 to take an image of the marker 751 until the collection of information necessary for the line-of-sight correction is completed (NO in S14). By doing so, the position and line-of-sight information of the two eye viewpoints when the operator stops the marker 751 (marker A) from the data of the operator captured by the camera 510 are obtained. Specifically, the marker A reflection point center 761 pA, the marker A pupil center 760 pA, the marker A arithmetic corneal curvature center KcAs, and the marker A arithmetic line-of-sight vector GsA are obtained.

When the collection of information necessary for line-of-sight correction is completed due to the stationary view of the marker 751 (YES in S14), the control unit 300 similarly causes the operator to stay at another marker 752 and captures it with the camera 510. Let In order to do so, the control unit 300 causes the display unit 410 to display the status "Gaze correction is in progress. Look at the marker on the upper right of the operation unit." (S15). The control unit 300 causes the operator to stay on the marker 752 until the information gathered for the eye gaze correction is completed by stopping the marker 752 on the basis of the data of the operator who stays on the marker 752 (marker B). Then, the image is taken by the camera 510 (NO in S16). By doing so, the marker B reflection point center 761pB, the marker B pupil center 760pB, the marker B arithmetic corneal curvature center KcBs, and the marker B arithmetic line-of-sight vector GsB are obtained from the data of the operator photographed by the camera 510.

Further, the control unit 300 makes the operator stop the other marker 753 and causes the camera 510 to take an image of it. In order to do so, the control unit 300 causes the display unit 410 to display the status "Gaze correction is in progress. Look at the marker at the lower right of the operation unit." (S17). The control unit 300 causes the operator to stay on the marker 753 until the collection of information necessary for the line-of-sight correction is completed based on the data on the operator who stays on the marker 753 (marker C). Then, the image is taken by the camera 510 (NO in S18). By doing this, the marker C reflection point center 761pC, the marker C pupil center 760pC, the marker C arithmetic corneal curvature center KcCs, and the marker C arithmetic line-of-sight vector GsC are obtained from the data of the operator photographed by the camera 510. The control unit 300 calculates the line-of-sight correction value based on the information necessary for line-of-sight correction thus obtained, and causes the display unit 410 to display the status “Line-of-sight correction is being calculated. Please wait.” (S19). The control unit 300 also turns off the camera position display unit 421 (S20). The control unit 300 proceeds with the process until the calculation of the line-of-sight profile information for each operator (calculation of the line-of-sight correction value) by the information creating unit 360 is completed, in other words, until the line-of-sight of the operator can be properly detected. Is waited (NO in S21). When the calculation of the line-of-sight correction value is completed, the control unit 300 (specifically, the line-of-sight detection unit 330) can substantially match the line-of-sight vector (see FIG. 4) and the address of the operation panel 700 (see FIG. 8). .. In the above description, the display prompting the user to see each of the markers 751 to 753 is sequentially displayed for each of the markers 751 to 753, but the display is not limited to this. For example, when an LED is provided as the target portion instead of the markers 751 to 753, a display such as “Please look at the LEDs that light up in order” may be displayed.

The calculation of the line-of-sight correction value will be described by taking the line-of-sight correction value of the marker 751 in the YZ direction as an example. To calculate the line-of-sight correction value, the center position coordinates “X=AA01x, Y=AA01y, Z=AA01z” of the marker 751, the marker A reflection point center 761pA, the marker A pupil center 760pA, the marker A arithmetic corneal curvature center KcAs, the marker The A arithmetic line-of-sight vector GsA is used. The marker A corneal curvature center KcA is obtained from the arithmetic line-of-sight vector GsA of the marker 751 and the Y coordinate “AA01y” and Z coordinate “AA01z” of the marker 751 on the operation panel 700. The coordinate deviation between the marker A corneal curvature center KcA and the marker A arithmetic corneal curvature center KcAs is an error between each operator and an average value using an arithmetic expression, and the line-of-sight measurement of the line-of-sight detection unit 330 is made the error as a correction value. Used for calculation formula (not shown). Higher visual line detection accuracy can be obtained by performing arithmetic on this in the XZ direction, and also by performing similar arithmetic operation on visual line information obtained by keeping the markers 752 and 753 stationary.

Returning to the description of FIG. 7, the control unit 300, based on the line of sight of the operator detected at each predetermined position where the markers 751 to 753 are provided by executing the information acquisition mode as described above, the line-of-sight profile for each operator. Create information. Then, when it becomes possible to accurately detect the line of sight based on the created line-of-sight profile information (YES in S21), the control unit 300 shifts to the line-of-sight input mode in which various functions are executed by the line of sight of the operator. The line-of-sight input mode is a mode in which an input relating to image formation is accepted according to the line-of-sight input operation by the user's line of sight detected in the setting mode. With the transition from the setting mode to the line-of-sight input mode, the control unit 300 displays “Status of completion of line-of-sight correction and start line-of-sight detection” on the display unit 410 (S22) to detect the line-of-sight of the operator. It starts (S23). As described above, in the present embodiment, the line of sight of the user is set in a state in which the line of sight of the user can be detected in the setting mode, and the line of sight input operation by the line of sight of the user detected based on the setting of the setting mode in the line of sight input mode becomes possible. This is notified to the operator. In other words, the user is notified that the setting to the state in which the user's line of sight can be detected in the setting mode is completed, or the user is notified that the setting mode has been changed to the line-of-sight input mode. At this time, the control unit 300 (specifically, the display control unit 341) operates the display unit 410 in order to notify the operator of the position on the display unit 410 at which the line of sight of the operator is facing, at the start of the detection of the line of sight. A line-of-sight pointer 810 (see FIG. 2) is displayed as a mark indicating the line of sight of the person (S24). Then, the control unit 300 starts line-of-sight input (S25). That is, the line-of-sight pointer 810 displayed on the display unit 410 is displayed so as to move on the display unit 410 in accordance with the line-of-sight movement of the operator. I can recognize what is happening. The operator can execute these operations, that is, various functions, by directing the line of sight to the hard key 401 of the hardware unit or by aligning the line of sight pointer 810 with the soft key 403 displayed on the display unit 410 by the line of sight. Can be done (line-of-sight input).

When the line-of-sight detection is possible, the line-of-sight detection unit 330 continuously calculates the position based on the data of the operator photographed by the camera 510. As a result, the line-of-sight pointer 810 displayed on the display unit 410 is displayed by the display control unit 341 so as to move on the display unit 410 in accordance with the movement of the line of sight of the operator. Then, when the line of sight of the operator deviates from the display unit 410 and moves to the hardware unit 400, the display control unit 341 hides the line-of-sight pointer 810. Further, if the operator's line of sight is directed to one of the hard keys 401 of the hardware unit 400, the display control unit 341 displays information (for example, name and function) on the hard key 401 of the line of sight on the display unit 410. .. In the specification of the line-of-sight destination by the display control unit 341, the coordinates of each area from the address “AA01” to “KK13” of the operation panel 700 (see FIG. 8) and the coordinates of the line-of-sight specified by the line-of-sight detection unit 330 substantially match. It depends on what you do. For example, since the address of the low power mode button of the hardware unit 400 is “BB12”, when the coordinates of the line of sight are substantially coincident with the coordinates of the address “BB12”, the status of “Low power button is displayed on the display unit 410 is displayed. Is displayed.” is displayed.

Then, the control unit 300 determines whether or not the presence sensor 450 is detecting the location of the operator (S26), and as long as the presence of the operator is detected (S26: (YES), and returns to the process of S25 to continue the line-of-sight input. On the other hand, when the state where the location of the operator is detected is changed to the state where the location of the operator is not detected (NO in S26), the control unit 300 ends the main line-of-sight input process. As described above, when the operator's line-of-sight detection does not require very high accuracy, the line-of-sight correction may not be performed. In this case, when the line-of-sight detection is completed in S10 of FIG. 6, a message such as "Line-of-sight detection completed. Input operation by line-of-sight is possible" is displayed on the display unit in S11. That is, when the line-of-sight correction is not performed, the setting mode is completed in S10. Then, after executing the process of S11 of FIG. 6, the processes of S23 and subsequent steps of FIG. 7 are executed. That is, when the line-of-sight correction is not performed, the line-of-sight input operation is possible based on the line-of-sight detected in S10. Further, whether or not to perform the line-of-sight correction may be set by the operator.

<Example of eye-gaze input operation>
The above-mentioned line-of-sight input operation will be specifically described with reference to FIGS. 9 and 10. In FIG. 9, the case where the operator operates the “copy button (COPY)”, which is the soft key 403 displayed on the display unit 410, by the line of sight will be described as an example. By checking the line-of-sight pointer 810 displayed on the display unit 410, the operator can move the line of sight onto the copy button. The address of the copy button is “CC02, CC03, DD02, DD03” (see FIG. 8). When the operator keeps the line-of-sight pointer 810 in the four areas for the first time or longer (for example, 2 seconds or longer), the line-of-sight detection unit 330 can determine that the operator is looking at the copy button. .. When the position of the line of sight of the operator is determined by the line-of-sight detection unit 330, the display control unit 341 causes the display unit 410 to display “Status such as “copy button selected”” as shown in FIG. In addition, the display control unit 341 changes the display color of the soft key (here, the copy button) that is overlapped with the line-of-sight pointer 810 to a color different from the display color of the other soft keys that are not overlapped with the line-of-sight pointer 810. This allows the operator to confirm whether or not the position of his own line of sight and the position of his line of sight detected by the image forming apparatus match.

After that, when the state of the line-of-sight pointer 810 within the above-mentioned four areas continues for a second time or longer (for example, 1 second or longer), the display control unit 341 switches the display of the display unit 410. For example, as shown in FIG. 10, the display control unit 341 causes the display unit 410 to display a selection screen having a “Yes” button and a “No” button as the soft key 403 instead of the “Copy button”. At the same time, the status of the display unit 410 is switched from “selecting the copy button” to “decide on copy button”. Then, when the operator looks at the “Yes” button, the function executing unit 350 operates the apparatus so as to execute the copy function assigned to the “Copy button”. On the other hand, when the operator turns his sight to the “No” button, the display control unit 341 performs control to return from the above selection screen to the screen (see FIG. 9) before the selection of the “copy button”. The first time and the second time may be the same time (for example, both are 1 second or more) as long as it is possible to determine the state in which the line-of-sight pointer 810 is located within the predetermined area for the predetermined time.

In the present embodiment, for example, when the operation method is unknown, the operator operates the “help” button displayed on the display unit 410 by line-of-sight input to call a manual or make an inquiry to the support center. It may be possible to perform operations other than image formation.

Further, in the above-described embodiment, when the hard key 401 or the soft key 403 is selected by the line of sight, information about the selected hard key 401 or the soft key 403 is displayed on the display unit 410 to notify the operator. However, it is not limited to this. For example, the hard key 401 is blinked, the display color or size of the soft key 403 is changed, or the like. The hard key 401 or the soft key 403 ahead of the operator's line of sight, that is, the hard key 401 operated by line-of-sight input. It suffices that the or soft key 403 be notified. Furthermore, you may notify by the sound (sound guide) using the speaker 430, or the vibration using the vibration generator 440.

It is preferable that the created line-of-sight profile information for each operator is stored in the memory 320 (see FIG. 5) so that the line-of-sight profile information can be called and used when the image forming apparatus 50 is used next time. By doing so, when the same operator uses the image forming apparatus 50 next time, the execution of the information acquisition mode (S12 to S21 in FIG. 7) can be omitted. For example, by providing the image forming apparatus 50 with an authentication device 800 (see FIG. 1) as a specifying unit and performing authentication with an ID card in which personal information is written, corresponding line-of-sight profile information can be read from the memory 320. You can do it. That is, when the operator specified by the authentication device 800 is the operator whose line-of-sight profile information is stored in the memory 320, it suffices to read the corresponding line-of-sight profile information from the memory 320 as the above-described information acquisition mode. In other words, when the operator specified by the authentication device 800 is an operator whose line-of-sight profile information is not stored in the memory 320, the above-described information acquisition mode is executed to create the line-of-sight profile information to create the memory 320. Remember.

As described above, in the present embodiment, the fact that the line-of-sight of the operator can be detected using the line-of-sight detection device 500 is notified to the operator by display, voice, vibration, or the like. Since the operator knows that the line-of-sight can be detected by the notification, the operator can immediately perform the operation by the line-of-sight input, which improves the convenience of the user. On the other hand, when it is not informed that the line-of-sight can be detected as in the conventional case, the operator does not know whether the operation can be performed by the line-of-sight input, and the operation by the line-of-sight input takes time. In particular, the operator who first inputs the line of sight needs to detect the line of sight as a preparation for that, and it is difficult to smoothly perform the line of sight input unless it is notified that the line of sight can be detected. In the present embodiment, by notifying the operator that the line-of-sight of the operator can be detected, the operator can smoothly perform the transition from the line-of-sight detection as preparation to the line-of-sight detection for line-of-sight input. Therefore, user convenience is improved. As described above, it is possible to realize a configuration in which the visual line detection technology is adopted in the image forming apparatus 50, which is more convenient for the operator, that is, more convenient for the user.

In the above-described embodiment, the operation panel 700 having the hardware section 400 and the display section 410 has been described as an example, but the present invention is not limited to this. For example, the operation panel 700 may have only the hardware section 400 or may have an entire liquid crystal panel having only the display section 410. That is, only the hard key 401 or only the soft key 403 displayed on the display unit 410 may be operable by the operator's line of sight. Further, in the above-described embodiment, the line-of-sight detection device 500 is provided on the operation panel 700, but the present invention is not limited to this. For example, the line-of-sight detection device 500 is provided in the device main body 50a, the document reading device 601, or the like at a position around the operation panel 700 and at a position where the line of sight of the operator with respect to the operation panel 700 can be detected. May be.

50... Image forming apparatus, 300... Control unit, 320... Storage means (memory), 330... Line-of-sight detection unit (line-of-sight detection unit), 341... Display control unit (display control unit), 350... Execution unit (function execution unit) 360... Information creating means (information creating unit), 401... Operator (hard key), 402... Notification means (light emitting means, LED for light emission), 403... Operator (soft key), 410... Notification means (display means) , Display section), 420... Notification means (light emitting means, display lamp), 430... Notification means (sound generation means, speaker), 440... Notification means (vibration generation means, vibration generation device), 500... Sight line detection means (line of sight) Detection device) 510... Imaging unit (camera), 700... Operation unit (operation panel), 710... Panel cover, 751, 752, 753... Target part (marker), 800... Identification means (authentication device)

Claims (12)

An image forming apparatus capable of forming an image on a recording material,
An operation unit having a user interface capable of input operation by the user's line of sight,
An imaging unit for imaging the user's face,
A setting mode in which the line of sight of the user whose face is captured by the image capturing unit is set in a detectable state, and image formation in accordance with a line-of-sight input operation on the user interface by the line of sight of the user detected based on the setting of the setting mode And a control unit capable of executing an input mode for receiving an input,
The control unit notifies the user that a line-of-sight input operation based on the user's line of sight in the input mode is possible.
An image forming apparatus characterized by the above. An image forming apparatus capable of forming an image on a recording material,
An operation unit having a user interface capable of inputting by the user's line of sight,
An imaging unit for imaging the user's face,
A setting mode in which the line of sight of the user whose face is captured by the image capturing unit is set in a detectable state, and image formation in accordance with a line-of-sight input operation on the user interface by the line of sight of the user detected based on the setting of the setting mode And a control unit capable of executing an input mode for receiving an input,
The control unit notifies the user that the setting to a state in which the user's line of sight can be detected in the setting mode has been completed,
An image forming apparatus characterized by the above. An image forming apparatus capable of forming an image on a recording material,
An operation unit having a user interface capable of inputting by the user's line of sight,
An imaging unit for imaging the user's face,
A setting mode in which the line of sight of the user whose face is captured by the image capturing unit is set in a detectable state, and image formation in accordance with a line-of-sight input operation on the user interface by the line of sight of the user detected based on the setting of the setting mode And a control unit capable of executing an input mode for receiving an input,
The control unit notifies the user that the setting mode has been changed to the input mode,
An image forming apparatus characterized by the above. The operation unit has a display unit capable of displaying a soft key touchable by a user,
In the input mode, the control unit can perform a line-of-sight input operation on the soft key according to the line-of-sight of the user.
The image forming apparatus according to claim 1, wherein the image forming apparatus is an image forming apparatus. The operation unit has hard keys operable by a user,
In the input mode, the control unit can perform a line-of-sight input operation on the hard key according to the line-of-sight of the user.
The image forming apparatus according to claim 1, wherein the image forming apparatus is an image forming apparatus. A panel cover provided to expose the display means,
The imaging unit is provided on the panel cover,
The image forming apparatus according to claim 4, wherein the image forming apparatus comprises: The imaging unit is provided in the operation unit,
The image forming apparatus according to claim 1, wherein the image forming apparatus is an image forming apparatus. A notification means for notifying the user,
The notification unit is provided in the operation unit,
The image forming apparatus according to claim 1, wherein the image forming apparatus is an image forming apparatus. The notifying means is a display means for notifying the user by display.
The image forming apparatus according to claim 8, wherein: The notifying unit is a light emitting unit that notifies the user by light.
The image forming apparatus according to claim 8, wherein: The notifying unit is a sound generating unit that notifies the user by sound.
The image forming apparatus according to claim 8, wherein: The notifying unit is a vibration generating unit that notifies the user by vibration.
The image forming apparatus according to claim 8, wherein:

Images