JPH06331880A - Eyeball(line of sight) movement input camera and eyeball(line of sight) movement input device - Google Patents

Abstract

PURPOSE:To issue an instruction to a device such as a camera even when information on the accurate position of the line of sight is not obtained by detecting an eyeball(line of sight) movement pattern without depending on a position seen with the eyeball based on output from a photoelectric conversion means. CONSTITUTION:A part of luminous flux passing through a photographing lens 1 is transmitted through the translucent part of a main mirror 2, bent downward by a sub mirror 3 and guided to a focus detecting optical block 6. The light reflected by the main mirror 2 forms an image on a finder mat 8. A device for detecting the movement of an eye is constituted of a light emitting part for illuminating the eye 13, a lens 14, an image-formation lens 11 forming the image of the eye and a photoelectric conversion element 12 receiving the image. By detecting the change of the direction of the line of sight or the time wise variation of the position of a parameter concerning the eyeball image on the photoelectric conversion element 12 where the eyeball image is formed, the movement and the changing state of the line of sight are discriminated, and the instruction in accordance with the discriminated result is issued to the camera.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eyeball (line-of-sight) movement input device for detecting an eye movement and giving an instruction to a device such as a camera based on the detected movement.

[0002]

2. Description of the Related Art Conventionally, there has been proposed a method of detecting a position where an eye is looking and selecting a specific area from a plurality of focus detection areas of a camera. In that case, the focus detection area at the position where the line of sight matches is selected from the plurality of focus detection areas. Further, it has been proposed to detect the position of the line of sight seen by the eye and give various instructions to a device such as a camera. In the latter case, the applicant of the present invention, for example, discloses in Japanese Patent Laid-Open No.
So, I propose a release with a line-of-sight instruction to prevent camera shake. In this case, when the line of sight of the photographer matches the set line-of-sight release position, the camera is released.

In these conventional methods, it has been premised that the position itself where the eye is looking is detected within the field of view of the finder.

[0004]

As a method of detecting the position (line-of-sight position) of the eye 15 with respect to the visual field frame of the finder screen shown in FIG. 6, the eyeball is illuminated and Purkinje, which is an image of the illumination light source, is detected. A method is proposed in which the position of the image (P in FIG. 5a) is detected, the position of the pupil center (obtained from the pupil boundary of D in FIG. 5a) is further detected, and the position of the line of sight is known from the positional relationship between the Purkinje image and the pupil center. Has been done. Figure 5a
And FIG. 5b correspond to the case where the line of sight is looking at a different position, respectively, and thus the relative positional relationship between the Purkinje image and the center of the pupil depends on the position where the line of sight is seen.
It is possible to detect which position the eye 15 is looking on the finder screen from the relative positional relationship between the two.

In a device such as a camera, since the relative position between the face and the device (camera) is not fixed, the line-of-sight position cannot be determined with a single type of parameter relating to the eyeball image.
For example, it is necessary to detect a plurality of types of parameters such as the Purkinje image and the center of the pupil as described above to eliminate the influence of the indefinite relative position of the face and the device. Therefore, the eye gaze position detection device has a high image analysis capability. However, there was a problem that the cost was increased.

Further, as a practical problem, in order to accurately determine where the line of sight is looking on the screen, it is not sufficient to obtain the line-of-sight direction using the above-mentioned plurality of parameters, and it is not enough to determine the eyeball for each individual. There is a problem that it is necessary to memorize the difference in characteristics in a device such as a camera in advance and calibrate the difference between the detected line-of-sight direction and the line-of-sight position actually seen.

It is therefore an object of the present invention to provide a method of giving an instruction to a device such as a camera even if there is no information regarding the exact position of the line of sight.

[0008]

To achieve the above object, the present invention uses a method that does not require detection of the position of the line of sight with respect to the field frame. That is, the present invention relates to claim 11.
As described in the above, "instruction means for giving a predetermined instruction in the function of a predetermined device, optical means for forming an image associated with an eyeball, and photoelectric conversion for receiving the image formed by the optical means. Means, a detection means for detecting an eyeball (line of sight) movement pattern that does not depend on the position of the eyeball based on the output of the photoelectric conversion means, and the instruction means is activated based on the output of the detection means. An eyeball (line-of-sight) movement input device characterized by having a starting means. "

[0009]

Action: That is, a change in the direction of the line of sight (the above-mentioned calibration is not necessary because the change in the direction of the line of sight is not the position where the line of sight is looking itself), or the eyeball on the photoelectric conversion element on which the eyeball image is formed. We focused on the point that it is possible to give instructions to a device such as a camera by finding the movement of the image.

In this method, it is not necessary to accurately obtain the line-of-sight position itself with respect to the finder field frame, but it is sufficient to detect the temporal change of the line-of-sight direction or the eyeball image. In order to obtain the change in the line-of-sight position, it is necessary to obtain the position of the line-of-sight as the first step. In order to obtain the position of the line of sight, a method is known in which the Purkinje image and the center of the pupil are obtained and the position of the line of sight is obtained from the relationship between the positions of the two, and it has already been commercialized. However, in order to obtain an accurate line-of-sight position, in order to correct the difference in the characteristics of the eyeball of the photographer, in advance, in relation to the characteristics of the eyeball of the photographer, the detected line-of-sight direction and the position where the eye actually looks It was necessary to register the difference in the above as calibration data in the camera. Therefore, there is a problem in that it is not easy to use.

However, since the present invention is a method of detecting a change in the line-of-sight direction, it is not necessary to obtain an accurate line-of-sight position, and calibration is unnecessary. Since the present invention requires only a change in the direction of the line of sight, the apparatus is further simplified to detect only a single type of parameter such as the Purkinje image or the pupil center or the white-eye-black-eye boundary, and these images are Detecting changes in the positions of these images on the formed photoelectric conversion means can be used as a substitute for detecting changes in the line-of-sight direction.

Further, since the exposed part of the eyeball is horizontally long, the pupil boundary in the vertical direction and the boundary of the white-eyed iris in the vertical direction are often hidden, and the pupil boundary and the white-eye are detected to detect the movement of the line of sight. The method that needs to detect the boundary between the eye and the black eye has a problem that it is often difficult to detect the movement of the line of sight in the vertical direction. However, if only the Purkinje image needs to be detected, such a restriction disappears.

In the present invention, a change in the direction of the line of sight or a temporal change in the parameter position relating to the eyeball image on the photoelectric conversion element on which the eyeball image is formed is detected, and the state of the movement / change is identified to obtain the identification result. It is possible to give a corresponding instruction to the camera or the like. According to this method, it is not necessary to accurately obtain the line-of-sight position with respect to the finder field frame (the position where the eyes actually look), and it is possible to give an instruction to the apparatus by the movement of the eyes.

As described above, according to the present invention, (1) in the case of detecting the temporal change based on the line-of-sight direction, (2) in the case of detecting the temporal change in the position of the parameter (feature point) regarding the eyeball image on the photoelectric conversion element. It can be roughly divided into In the following description of the embodiments, the case (2) will be mainly described. Of course, the line-of-sight direction obtained by a known method instead of the position (parameter) of the eyeball image on the photoelectric conversion element in the following description. It is possible to correspond to the case of (1) by using.

[0015]

1 is a diagram showing a block configuration of an embodiment of the present invention. A part of the light flux that has passed through the taking lens 1 passes through the semi-transparent portion of the main mirror 2, is bent downward by the sub mirror 3, and is guided to the focus detection optical block 6. A perspective view of the structure of the focus detection optical block 6 is shown in FIG. 2, which includes a field mask 210, a field lens 230, a re-imaging lens mask 240, a re-imaging lens 250, and an image sensor array 260. It has a well-known configuration of the detection device. The focus detection circuit is the control circuit 10
Based on the instruction No. 3, the focus detection result regarding the designated focus detection area is output to the control circuit. Based on this, the control circuit causes the lens driving circuit 107 to focus the photographing lens 1. Since such a focus detection device is conventionally known, it will not be described in detail.

The light reflected by the main mirror forms an image on the finder matte 8. The focus detection area display circuit 9 can display the focus detection area as shown in FIG. 6, and normally, only the selected area is displayed so as to be distinguishable from other areas. Specifically, methods such as using liquid crystal, using electrochromic (EC), and illuminating with an LED are known and will not be described in detail here.

The device for detecting the movement of the eye comprises a light emitting section 13 for illuminating the eye, a lens 14, an imaging lens 11 for forming the image of the eye, and a photoelectric conversion element 12 for receiving the image of the eye. The viewfinder screen is surrounded by the visual field frame as shown in FIG. 6, and the conventional visual axis detection used for the camera detects which position the eye 15 is looking at with respect to the visual field frame. In the embodiment, since the movement of the eye is detected, it is not necessary to detect which position the user is looking at.

In order to detect the movement of the eye, the photoelectric conversion element 12 of FIG. 1 is composed of a large number of pixels 2 as shown in FIG. 3a.
-Dimensional image sensor (a one-dimensional image sensor may be used when detecting only one-dimensional direction) or Fig. 3b.
PSD of can be used. There are various ways to read the position of the image feature related to the eye. FIG. 4a shows the appearance of an image of the eye formed on the photoelectric conversion element. When the image sensor of FIG. 3a is used as the photoelectric conversion element, the position (Xi, Yi) of the Purkinje image P, which is the image of the illumination light source, on the photoelectric conversion element can be read by image processing as well known, and the pupil can be read. The boundary D can be detected to read the position of the center of the pupil, or the boundary of the white and black eyes may be detected.

When a PSD is used as the photoelectric conversion element, the center of gravity of the light amount distribution of the image is obtained, and it is not possible to extract the image features individually as in the case of the image sensor, but the Purkinje image is different. Since the amount of light is overwhelmingly larger than that of the image pattern, the determination of the center of gravity of the distribution of the amount of light in the entire image is substantially equivalent to the determination of the position of the Purkinje image.

In this case, the photocurrents from the respective output terminals of FIG. 3b are respectively IX1, IX2 in the horizontal direction and IY1, in the vertical direction.
If IY2, the coordinates of the center of gravity of the light amount distribution (Xi, Yi)
Is given by Xi = (IX1-IX2) / (IX1 + IX2) Yi = (IY1-IY2) / (IY1 + IY2).

As described above, since the positional relationship between the camera and the eye is not fixed and there are individual differences, the coordinate position thus obtained cannot correspond to the position of the line of sight. Since the position of the line of sight is unnecessary and only the temporal change of the line of sight needs to be taken into consideration, when the position on the photoelectric conversion element for the feature relating to the image of the eyeball is detected as described above, Fig. 4c is the movement of the line of sight like 4b.
(More accurately, the movement of the eyeball) is required.

In the flowchart of FIG. 10, detection is started in step 1. The detection is started by turning on the camera power.
It may be based on N, or may be based on an operation of a specific member such as half-pressing of the release button after the power is turned on. Figure 1
The block indicated by 20 in the flowchart of 0 corresponds to the detection circuit 101 of FIG. 1, and in step 2, from the output of the photoelectric conversion element, the position (Xi, Yi ) Is detected.

In step 3, as shown in Table 1, each time t1, t2, t3, t4. ． Value and the position at each time R1 = (X1, Y1), R2
= (X2, Y2), R3 = (X3, Y3), R4 = (X
4, Y4). ． Is memorized. Of course, the time here is the time interval Δt1 (blank), Δt2 = t2-t1, and Δt3 =
t3-t2, Δt4 = t4-t3 ,. ． There is no substantial difference even if it is replaced with and stored.

[0024]

[Table 1] In step 4, the movement of the eyeball is detected based on this data. Various detection methods can be considered, but a new time tN and an old time tP are selected, and the positions at the respective times are RN = (XN, YN), RP = (XP, YP), and the new position RN is changed to the new position RN. A vector MVE = (XN-XP, YN-YP) is calculated, and the magnitude L = ABS (MVE) of this vector and the direction Θ (MVE) are calculated by a known method in vector calculation.

In this way, the size L and the direction Θ of the eye movement are found at each time. As a method of selecting the old and new times, two consecutive times are used when the detection time interval is long. For example, the movement detection at time t3 is repeated according to the results of times t3 and t2, and the movement detection at time t4 is repeated according to the results of times t4 and t3 (loop 51).

When the detection time interval is short and long, two time data with one or more data interposed therebetween are used, or data with a predetermined time interval is used. For example, in the pattern detection at the time t3, the results at the times t3 and t1 are used, and at the pattern detection at the time t4, the times t4 and t2.
Depending on the result of (1), the same process is repeated. The movement at each time as a result of the above calculation is shown in FIG. 11b.
It is preferable to store it in the movement memory 41. The stored contents are, for example, L at each time,
As the value of Θ,

[0027]

[Table 2] become that way. The eye movements detected in this manner are subjected to pattern classification in step 5.

FIG. 8A shows the case where the Purkinje image P hardly moves, that is, the Purkinje image P moves only by a predetermined value r or less. Further, FIG. 8b shows a case where the Purkinje image P moves more than the predetermined value r. As the classification, for example, the size L of the eye movement (see FIG.
When the movement amount of the Purkinje images P of a and 8b is compared with a predetermined value r, when L> r, it is determined that there is movement and the direction Θ is the direction of the eye movement, and if not so, there is no eye movement.

[0029]

[Table 3] In step 6, an instruction is given to the function of the camera according to the classification A, B, C, D.

A first example of instructions to the camera is shown below.

[0031]

[Table 4] In step 7, the control circuit 103 controls the camera based on the instruction content.

The same instruction content may be given to a plurality of detection patterns as follows.

[0033]

[Table 5] In this way, it is possible to give an instruction to the camera by eye movement, but since the line of sight freely moves in the viewfinder field, the above-mentioned method may give an instruction to the camera at an unintended time.

A second embodiment for solving this problem is shown in FIG.
1a will be described. In this example, a timing detection interrupt circuit 8 is further provided. The timing input circuit 104 in FIG. 1 is an operation member of the camera, and an interrupt for timing detection is entered by an operation such as a function key operation or a half-press of a release button. There are various methods of timing interrupt. One method is to rotate the eye movement detection cycle according to the flow 81 in FIG. 11a only at the moment when the timing interrupt occurs.
The function instruction is to be given only once at the beginning, only for a predetermined number of cycles, or within a limited time range within a predetermined time.

As another method, the detection cycle is always kept on, while the timing interrupt timing is shown in FIG.
It is transmitted to the function instructing unit in the flow 82 of 1a, and the function instructing unit gives the function instruction only for a predetermined time immediately after the interruption. In the case of this method, as shown in FIG. 11b, pattern storage for storing the past detection result of the eye movement pattern is performed, and the detection result as shown in Table 6 is left, so that a predetermined time immediately before the interrupt time is reached. It is also possible to give a function instruction based on the result of (1), or to give a function instruction based on the result within a predetermined time including the interruption time point. In particular, the latter method, in which the reading timing instruction is issued after or while performing the eye movement based on the intention, is ergonomically natural and preferable.

[0036]

[Table 6] A further improved example of the pattern classification circuit of step 5 of FIG. 11a will be described. Using the plurality of movement combinations stored for each time in step 41 of FIG. 11b, in step 5, when the movement combinations relating to these plurality of times satisfy a predetermined condition, they are classified into category A, and When a predetermined condition is satisfied, the object is classified into the category B, and the like, so that a complicated eye movement pattern can be detected, and the intention of the photographer can be transmitted more accurately.

Such an operation requiring a complicated combination of eye movements can be used, for example, for locking or unlocking the operation of the camera. Here, unlocking the operation of the camera means that at least a part of the functions of the camera does not function normally unless a predetermined movement of the line of sight is input (for example, release cannot be performed). This can prevent others from being used for shooting. In order to set the movement of the line of sight necessary for unlocking, the line of sight is moved and registered while pressing a predetermined function key or a plurality of combinations thereof.

The lock is released by moving the line of sight while pressing another function key or a plurality of combinations other than the combination. When the camera is turned off and then turned on, it will be locked. Of course, the lock may be performed by the same line of sight as in the unlocking.

In the case of FIG. 11a, step 5 or both steps 4 and 5 are outside block 20 and step 6
It may be arranged immediately before the above step, and after receiving the instruction 82 from the timing detection 8, step 5 or both steps 4 and 5 may be calculated. Next, an example of the detection pattern classification and the instruction content based on the result will be additionally described. FIG. 9 shows a multi-area focus detection optical system different from that shown in FIG. This is a field mask 91, a field lens 92, a re-imaging lens mask 93, a re-imaging lens 94, an image sensor array 95.
It is a multi-area focus detection optical system. The corresponding focus detection area on the viewfinder screen is shown in Fig. 7.
As shown in, the center, right, upper, left, and lower five points are provided. The focus detection area is selected by moving the eyeball as follows.

In the flow of FIG. 11a, the loop portion indicated by 20 blocks is repeatedly detected. By pressing the release button on the camera halfway, the timing of reading the eye movement is instructed in step 8.
Then, in step 6, a function instruction is given according to the following table.

[0041]

[Table 7] In step 7, the corresponding control of the camera is made. That is, the selected area is displayed and the focusing operation is performed on the selected area.

As another example of the instruction content, there is a release by the line of sight, which makes it possible to prevent camera shake that is likely to occur when the release button is fully pressed. Since it is inevitable that the line of sight moves around the screen in various ways,
It would be inconvenient if such a movement would easily release the shutter. Therefore, the condition of the line-of-sight release is that L> rbig holds for a large predetermined value rbig and that the direction Θ is within a predetermined angle range of the predetermined direction.

The predetermined direction may be, for example, the downward direction and the angular range may be, for example, plus or minus 10 degrees, but it may be set according to the preference of the photographer. Further, the attitude detection circuit may be built in the camera and the condition may be set on condition that the camera is in a predetermined direction with respect to the gravity direction regardless of the horizontal position or the vertical position.

In the above embodiment, the example in which the eye movement pattern is directly detected from the position information of the eye image has been described.
The gaze direction may be calculated once, and then the gaze movement pattern may be detected by obtaining the temporal change in the gaze direction. FIG. 12A is written corresponding to the case of calculating the line-of-sight direction in the case of FIG. 11A described above.

The difference between the two lies in steps 2, 3, and 4.
In order to obtain the line-of-sight direction, it is not enough to obtain the position of the Purkinje image, and another parameter, such as the position of the center of the pupil, is also read in step 2. In step 3, the Purkinje image position and the pupil center position are stored. Figure 4 in step 4
Step 44 of detecting the line-of-sight direction as in 2b.
Since a method of detecting the line-of-sight direction using the Purkinje image and the center of the pupil is known, it is omitted here.

The line-of-sight direction thus obtained is stored for each time (step 43), and the line-of-sight movement is detected by using the line-of-sight direction data for a plurality of hours in the same manner as when the eye-ball movement is detected from the eyeball images of the plurality of hours. Detection (step 42) and the result is stored (step 41). Although the drawing of the single-lens reflex camera has been used in the above description, the present invention can be similarly applied to the lens shutter camera as shown in FIG.

[0047]

As described above, according to the present invention, the eyeball (visual line) movement pattern which does not depend on the position of the eyeball is detected and the camera is instructed. With a simple structure of the detection device, it is possible to give an appropriate instruction to the camera by the movement of the eye.

In the case of the method of obtaining the eyeball (line-of-sight) movement pattern from the change of the line-of-sight direction, the conventionally required calibration is unnecessary or the accuracy of the calibration is low as compared with the case of detecting the known line-of-sight position itself. In the case of the method of obtaining the time change of the feature points of the eyeball image on the photoelectric conversion element, there is an advantage that the apparatus is greatly simplified.

According to the present invention, it is possible to give an instruction based on the movement of the eye to the camera in such a simplified format.

[Brief description of drawings]

FIG. 1 is a block diagram of an eyeball (gaze) input movement camera that is an embodiment of the present invention.

FIG. 2 is a perspective view of an optical system of a focus detection block.

FIG. 3a is an explanatory diagram when a two-dimensional image sensor for detecting a line of sight is used. FIG. 3b is an explanatory diagram when a PSD is used as a sensor for detecting the line of sight.

FIG. 4a is an explanatory diagram showing a state of a detected image of a line of sight when the sensor of FIG. 3a is used. 4b and 4c
FIG. 4 is an explanatory diagram showing a state of a detected image of a line of sight when the sensor of FIG. 3b is used.

FIG. 5a is a diagram showing a Purkinje image of an eyeball. FIG. 5b is a diagram showing a Purkinje image of the eyeball.

FIG. 6 is a diagram showing a display example of a field frame on a finder screen.

FIG. 7 is a diagram showing a display example of a field frame on a finder screen.

8A and 8B are diagrams for explaining how the Purkinje image moves.

FIG. 9 is a perspective view of an optical system of a focus detection block different from that of FIG.

FIG. 10 is a flowchart of eyeball (line-of-sight) movement detection.

FIG. 11a is a flow chart of eyeball (line of sight) movement detection. FIG. 11b is an explanatory diagram of the storage step of the eyeball movement detection result. Figure 11c
FIG. 7 is an explanatory diagram of a detection pattern classification step.

FIG. 12a is a flow chart of eyeball (line of sight) movement detection. FIG. 12b is an explanatory diagram of the storage step of the eyeball movement detection result.

FIG. 13 is an explanatory diagram when the present invention is applied to a lens shutter camera.

[Explanation of symbols]

 11, 12 Eyeball (line of sight) detection photoelectric conversion unit 13, 14 Eyeball (line of sight) detection light emitting unit

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁵ Identification code Internal reference number FI technical display location G03B 13/02 9120-2K 9119-2K G03B 3/00 A

Claims (11)

[Claims] 1. A photographing lens for forming an image of a subject on a predetermined image plane, an image recording means provided on the predetermined image plane, and an image formed on the predetermined image plane by the photographing lens. Finder means for guiding a related scene to the eyeball through the inside of a predetermined field frame, instruction means for instructing the function of a device such as a camera, optical means for forming an image related to the eyeball, and the optical means. Photoelectric conversion means for receiving the captured image, detection means for detecting an eyeball (line of sight) movement based on the output of the photoelectric conversion means, and activation means for activating the instruction means based on the output of the detection means An eyeball (line-of-sight) movement input camera characterized by having. 2. The camera according to claim 1, wherein the photoelectric conversion means is a PSD. 3. The camera according to claim 1, wherein the photoelectric conversion means is a two-dimensional image sensor. 4. The position detecting means for reading the position relating to the feature of the image of the eye formed on the photoelectric conversion means, the position storing means for storing the read position, and the plurality of stored means. 2. The camera according to claim 1, further comprising an eyeball movement detection unit that detects movement related to the eyeball based on a change in the characteristic position with respect to time. 5. The detecting means relates to a position reading means for reading a position relating to a feature of an image of an eye formed on the photoelectric conversion means, a position storing means for storing the read position, and a plurality of time points. 2. The camera according to claim 1, further comprising a line-of-sight movement detecting means for determining a line-of-sight direction and detecting movement in the line-of-sight direction from the change. 6. The camera according to claim 1, wherein the instructing means issues an instruction to start or end a predetermined operation in the function of the camera having the above configuration. 7. The camera according to claim 1, wherein the instructing means selects and instruct a specific function from among a plurality of functions among the functions of the camera having the above configuration. 8. The camera according to claim 1, wherein the instructing means issues an instruction to lock or unlock an operation of the camera having the above structure. 9. The detecting means further comprises a categorizing means for categorizing a plurality of eyeball (line-of-sight) movement patterns, and the indicating means has a function of a camera according to the result of categorizing by the categorizing means. 2. The camera according to claim 1, wherein a plurality of instructions such as an instruction to start or end a predetermined operation in step 1 or a selection of a specific function from among a plurality of functions in the camera is issued. 10. The detecting means further includes a categorizing means for categorizing a plurality of eyeball (line of sight) movement patterns, and the instructing means provides the same instruction to a plurality of categorizing results by the categorizing means. Instructions for starting / ending / selecting predetermined functions)
The camera according to claim 1, wherein at least one of the above is performed. 11. An instruction means for giving a predetermined instruction in the function of a predetermined device, an optical means for forming an image associated with an eyeball, and a photoelectric conversion means for receiving the image formed by the optical means. Detection means for detecting an eyeball (line of sight) movement pattern that does not depend on the position of the eyeball based on the output of the photoelectric conversion means; and activation means for activating the instruction means based on the output of the detection means. An eyeball (line-of-sight) movement input device characterized by having.