JP2016099959A - Direction indication device and direction indication method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a direction indication device which operates a screen of a computer through a head or horizontal/vertical motion of eyes of a user, and a direction indication method.SOLUTION: A direction indication device comprises: a camera (imaging means) 3 for capturing an image of a face area of an operator; first coordinate system setting means 5 and second coordinate system setting means 6 for setting a first coordinate system of the face area imaged by the imaging means and a second coordinate system of an eye area in the face area; first delay means 7 and second delay means 8 for, in association with the movement of the eye area, delay-processing an original point coordinate of the second coordinate system moving while delaying following that, and for obtaining third and fourth coordinate systems different in delay time; first calculation means 9 and second calculation means 10 for calculating a movement amount and movement speed of an iris center in the third and fourth coordinate systems; discrimination means 11 for performing operation discrimination based on distance between original points of the third and fourth coordinate systems; and first to fourth determination means 12, 13, 14 and 15 for determining a horizontal direction or a vertical direction by a head or an eyeball on the basis of a signal from the discrimination means.SELECTED DRAWING: Figure 2

Description

According to the present invention, a face image of a computer operator is continuously acquired by an imaging device built in the computer, and based on the up / down / left / right rotational movement of the head or eyeball, the computer is instructed to the right direction of the screen or The present invention relates to a direction indicating device and a direction indicating method for instructing a left direction, an upward direction, or a downward direction.

As a method for moving a cursor or the like in a computer screen, a mouse, a finger, and a voice are widely used. Recently, an input device that captures the movement of the head or eyeball with a camera and indicates the position according to the movement is also known (Patent Document 1).

Patent Document 1 discloses an invention in which the center of eyeball rotation is determined from feature points such as a cornea reflection image of near-infrared light and a branching point of capillaries on the surface of the eyeball by three-point target gaze. This prior invention is a structure for calculating the line-of-sight direction, and the line-of-sight measurement accuracy is extremely lowered depending on the position of the cornea reflection image. There are problems that feature point extraction is difficult, eye-gaze measurement accuracy is reduced due to individual differences in eyeball shape, and that the distance between the eyeball and the target is required to determine the center of eyeball rotation.

In addition, based on the method of estimating the eyeball rotation center by three-point gaze without personal parameters, it is necessary to estimate the gaze point from the pupil center position when gaze at least three targets on the plane to be viewed in order. A line-of-sight measurement method for determining parameters is also known (Patent Document 2). However, in the invention disclosed in this publication, the positioning accuracy is not sufficient, and there is room for improvement.

JP 2012-29940 JP 2014-52758 A

The present invention has been made in view of the above-described circumstances, and continuously obtains an operator's face image with a camera built in a computer, and appropriately performs low-pass filtering processing on an image including an eye region. Thus, an apparatus is provided that accurately transmits an instruction direction to a computer by a vertical or horizontal rotational movement of a head or an eyeball. In particular, the direction indication accuracy is improved by utilizing the delay characteristic (phase characteristic) of the low-pass filter LPF.

The direction indicating device according to the present invention includes an imaging means for imaging a face area of a computer operator, a first coordinate system setting means for setting a first coordinate system of the face area imaged by the imaging means, and the imaging means. To obtain a second coordinate system setting means for setting a second coordinate system of the eye area in the face area imaged in accordance with the above, and to obtain a third coordinate system that moves in a delayed manner following the movement of the eye area. 1 delay means, second delay means for obtaining a fourth coordinate system that moves with a delay following the movement of the eye area and has a delay time smaller than that of the first delay means, and the third coordinates First calculating means for calculating the moving amount and moving speed of the iris center in the system, second calculating means for calculating the moving amount and moving speed of the iris center in the fourth coordinate system, and the third coordinate system; Discrimination that performs operation discrimination based on the distance between the origins of the fourth coordinate system First determination means for determining a left / right direction instruction by a head based on a signal from the determination means, second determination means for determining a left / right direction instruction by an eyeball based on a signal from the determination means, and the determination A third determination unit that determines a vertical direction instruction from the head based on a signal from the unit; and a fourth determination unit that determines a vertical direction instruction from the eyeball based on a signal from the determination unit.

In the direction indicating device according to the present invention, the first delay means includes a first low-pass filter that performs low-pass filter processing on the origin coordinates of the second coordinate system, and the second delay means includes the second coordinate system. The origin coordinates may be a second low-pass filter that performs low-pass filter processing and has a delay time shorter than that of the first low-pass filter.

The direction indicating method according to the present invention includes a step of imaging a computer operator's face area, a step of setting a first coordinate system of the face area imaged by the step, and an eye area image of the imaged face area A step of setting a second coordinate system, a step of obtaining a third coordinate system that moves in a delayed manner following the movement of the eye region, and a step of following the movement of the eye region. Obtaining a fourth coordinate system that moves with a delay and having a delay time smaller than that of the third coordinate system; calculating a moving amount and a moving speed of an iris center in the third coordinate system; and the fourth coordinate Calculating a moving amount and moving speed of the iris center in the system, performing a motion determination based on a distance between the origins of the third coordinate system and the fourth coordinate system, and left and right by the head based on the motion determination Direction A step of determining a left / right direction instruction by an eyeball based on the motion determination; a step of determining a vertical direction instruction by a head based on the motion determination; and a determination of a vertical direction instruction by an eyeball based on the motion determination And steps to
It is made up of.

In the direction indicating method according to the present invention, the step of obtaining the third coordinate system is performed by a first low-pass filter that performs low-pass filter processing on the origin coordinates of the second coordinate system, and the step of obtaining the fourth coordinate system includes: The origin coordinate of the second coordinate system may be subjected to a low-pass filter process, and a second low-pass filter having a delay time shorter than that of the first low-pass filter.

According to the present invention, the operator moves his / her head or eyeball up / down / left / right while looking at the top / bottom / left / right marks on the computer screen or when the mark is not displayed. By exercising, the instruction direction can be accurately transmitted to the computer, and the screen can be scrolled in the vertical and horizontal directions according to the instruction. A disabled person who cannot use both hands or a person who cannot use both hands can operate the screen regardless of the keyboard or mouse.

It is a perspective view for demonstrating the use condition of the direction indicator which concerns on this invention. It is a block diagram which shows the structure of the direction indication apparatus which concerns on this invention. It is an image figure of the camera image which image | photographed the user's face. It is a figure for demonstrating face area extraction by OpenCV. It is a figure for demonstrating extraction of the eye area | region by OpenCV. It is a figure for demonstrating the eye area | region extraction by LPF. It is a figure for demonstrating the difference of the eye area | region at the time of head rotation. It is a figure for demonstrating the positional relationship of the eye area | region at the time of head rotation. It is a figure for demonstrating the positional relationship of the eye area | region at the time of eyeball rotation. It is a figure for demonstrating the movement of the iris at the time of head rotation. It is a figure for demonstrating the movement of the iris at the time of eyeball rotation. It is a graph showing the movement amount xe of the left-right direction of the iris center in head rotation motion (right instruction | indication). It is a graph showing the moving speed xd of the left-right direction of the iris center in head rotation movement (right instruction | indication). It is a graph showing the moving amount xe of the left-right direction of the iris center in head rotation movement (left instruction | indication). It is a graph showing the moving speed xd of the left-right direction of the iris center in head rotation movement (left instruction | indication). It is a graph showing the movement amount xe of the left-right direction of the iris center in an eyeball movement (right instruction | indication). It is a graph showing the moving speed xd of the left-right direction of the iris center in eyeball rotation movement (right instruction | indication). It is a graph showing the moving amount xe of the left-right direction of the iris center in an eyeball rotational movement (left instruction | indication). It is a graph showing the moving speed xd of the left-right direction of the iris center in eyeball rotation movement (left instruction | indication). It is a flowchart which shows the pre-processing part of the direction instruction | indication method which concerns on this invention. It is a flowchart which shows the process part for determination of the left-right direction instruction | indication of the direction indication method which concerns on this invention. It is a flowchart which shows the process part for determination of the up-down direction instruction | indication of the direction indication method which concerns on this invention.

In FIG. 1, 1 is a computer, 2 is an operator (hereinafter referred to as a user) of the computer 1, and 3 is a photograph of the head and face of the user 2 by a camera (imaging means) provided in the upper center of the screen of the computer 1. Shoot. From the image of the camera 3, two-dimensional coordinates of the iris center of the eye, that is, the eyeball are obtained, and the vertical and horizontal directions designated by the user 2 are determined from the movement amount.

Reference numeral 4 denotes five marks set in the screen of the computer 1, which are arranged at the center, left and right, and top and bottom of the screen. For example, the user 2 moves the line of sight (indicated by a broken line) to the mark 4 on the right side after moving the line of sight to the mark 4 located at the center and then returns to the center mark 4 to scroll the screen to the right. The screen can be scrolled to the left by moving the line of sight to the left mark 4 and then returning to the center mark 4 again. Similarly, if the line of sight is moved from the central mark 4 to the upper or lower mark 4, the screen can be scrolled up or down. The same operation can also be realized by rotating the head up and down, left and right in the direction of the mark without moving the line of sight. In practice, the same operation can be performed by moving the line of sight up and down or rotating the head of the screen without displaying the mark 4 on the screen.

In this way, the screen can be scrolled up and down and left and right by moving the line of sight or rotating the head. When moving the screen to the left and right, for example, a page of a book is displayed on the screen, and the movement of the head or line of sight is displayed. It is possible to perform operations such as turning pages. The computer 1 includes a notebook computer, a tablet, a smartphone, and the like.

The following parameters are used in the configuration of the direction indicating device according to the present embodiment.
αx, αy: Threshold values for determining whether the movement of the iris center is caused by head rotation or eyeball rotation. If these values are set small, the direction of the head is instructed with priority given to the movement of the head over the eyeball. On the other hand, when the setting is large, the direction is instructed with priority given to the movement of the eyeball. This parameter determines the priority of the instruction operation.
βx1, βy1: Threshold values for the movement amount of the iris center used when determining whether or not the user has directed the direction with the head. If these values are set small, the direction can be instructed to the computer even if the head is moved slightly. However, the frequency of unintended direction indication increases accordingly. Therefore, when it is desired to give an instruction only when the head is moved largely, these values are set large.
βx2, βy2: Threshold values for the moving speed of the iris center used when determining whether or not the user has directed the direction with the head. If these values are set to a small value, it is possible to indicate the direction even if the head is moved slowly. However, the frequency of unintended instructions increases accordingly. Therefore, when it is desired to give an instruction only when the head is moved quickly, these values are set large.
βx3, βy3: Threshold values for the movement amount of the iris center used when determining whether or not the user has directed the direction with the eyeball. If these values are set to be small, it is possible to indicate the direction even if the eyeball is moved slightly. However, the frequency of unintended direction indication increases accordingly. Accordingly, when it is desired to give an instruction only when the eyeball is moved largely, these values are set large.
βx4, βy4: Threshold values for the moving speed of the iris center used when determining whether or not the user has directed the direction with the eyeball. If these values are set small, it is possible to indicate the direction even if the eyeball is moved slowly. However, the frequency of unintended direction indication increases accordingly. Therefore, when it is desired to give an instruction only when the eyeball is moved quickly, these values are set large.
Lx, Ly: Data length used for discriminating head and eye movements. If this value is set large, the direction is indicated only when the head and the eyeball are moved over a relatively long time. If you want to avoid direction indications due to unintended head and eye movements, set these values large.
Nx, Ny: Data length used in direction indication judgment. If this value is set large, the direction is indicated only when the head and the eyeball are moved over a relatively long time. If you want to avoid direction indications due to unintended head and eye movements, set these values large.
Mx, My: Data length used for direction indication judgment. If this value is set to a large value, the direction is indicated only when the head and eyeball are moved largely over a relatively long time. If you want to avoid direction indications due to unintended head and eye movements, set these values large.

    In FIG. 2, 3 is the above-mentioned camera which images the user's 2 eyes and face area | region, and the image figure of the imaged image is shown in FIG. Reference numeral 5 denotes a first coordinate system setting means for setting the first coordinate system 0 of the face area imaged by the camera 3, and uses an existing library such as OpenCV (Open Source Computer Vision Library) to detect the face from the camera image. A region is extracted, and a first coordinate system O (X axis, Y axis) with the upper left corner as the origin is set (FIG. 4). Reference numeral 6 denotes second coordinate system setting means for setting the second coordinate system O ′ of the eye area in the image captured by the camera 3. Similarly, the eye area is determined from the face area using an existing library such as OpenCV. Extract and set the second coordinate system O ′ (Xo axis, Yo axis) with the upper left corner as the origin (FIG. 5).

Reference numeral 7 denotes a first delay means for obtaining a third coordinate system O1 which moves the origin coordinate of the second coordinate system O ′ following and delaying the movement of the eye region. The origin coordinates of 'can be configured by a first low-pass filter LPF1 that performs low-pass filter processing. Similarly, the fourth coordinate system O2 moves the origin coordinate of the second coordinate system O ′ following the delay of the eye region as the eye region moves, and has a delay time smaller than that of the first delay means 7. The obtained second delay means can be constituted by, for example, a second low-pass filter LPF2 having a delay time smaller than that of the first low-pass filter LPF1, which performs low-pass filter processing on the origin coordinates of the second coordinate system O ′.

The relationship between the first and second low-pass filters LPF1 and LPF2 is that the first low-pass filter LPF1 is a low-pass filter having a large delay time (low-pass filter having a low cut-off frequency), and the second low-pass filter LPF2 is a low-pass filter having a small delay time. (A low-pass filter with a high cut-off frequency). FIG. 6 shows eye region extraction by the first and second low-pass filters LPF1 and LPF2, and FIG. 7 shows the difference between eye regions during head rotation. The cutoff frequencies of the first and second low-pass filters LPF1 and LPF2 can be set to 4.5 Hz and 10 Hz, for example.

If the cutoff frequencies of the first and second low-pass filters LPF1 and LPF2 are set in such a relationship, when the user 2 rotates his / her head, the moving speed of the third coordinate system O1 is as shown in FIG. , Later than that of the fourth coordinate system O2. When processing with the first low-pass filter LPF1, the rectangular frame indicating the eye region moves smoothly and slowly (by moving slowly, direction indication detection by head rotation is possible), while when processing with the second low-pass filter LPF2 The square frame (fourth coordinate system O2) moves relatively quickly. FIG. 8 shows the positional relationship of the eye regions when the head rotates, and FIG. 9 shows the positional relationship of the eye regions when the eyeball rotates.

Reference numeral 9 denotes a first calculation means for calculating the moving amount and moving speed of the iris center in the third coordinate system O1, and the third coordinates using image binarization, noise removal processing by expansion and contraction, and labeling processing technology. For the system O1, the x1 coordinate value x1c and the y1 coordinate value y1c of the iris center are obtained.

Reference numeral 10 denotes a second calculation means for calculating the moving amount and moving speed of the iris center in the fourth coordinate system O2. Similarly, the second calculating means 10 uses the binarization of the image, the noise removal processing by expansion and contraction, and the labeling processing technique. Also for the four coordinate system O2, the x2 coordinate value x2c and the y2 coordinate value y2c of the iris center are obtained.

11 is a discriminating means for discriminating movement based on the distance between the origins of the third coordinate system O1 and the fourth coordinate system O2. A low cut filter is applied to the value x1c of the x1 coordinate and the value y1c of the y1 coordinate, And ye1. As a result, the bias components of the values x1c and y1c are removed, and the movement amount of the iris in the left-right or vertical direction is obtained. Further, a differential filter is applied to the values x1c and y1c, and the values are set as values xd1 and yd1, respectively. Thereby, the moving speed of the iris in the horizontal direction and the vertical direction can be obtained. Similarly, a low cut filter and a differential filter are applied to the value x2c of the x2 coordinate and the value y2c of the y2 coordinate to obtain the value xe2, the value xd2, the value ye2, and the value yd2, respectively.

Further, a deviation Dx = | Xo1-Xo2 | in the x-axis direction between the origins of the third coordinate system O1 and the fourth coordinate system O2 is calculated. If the state of Dx ≧ αx occurs continuously for Lx samples, set the movement amount xe = xe1 and the movement speed xd = xd1 (after that, move to the process when the head is directed left and right), and so on If not, the movement amount xe = xe2 and the movement speed xd = xd2 are set (after that, the process proceeds to the case where the left and right directions are instructed by the eyeball). Similarly, the deviation Dy = | Yo1-Yo2 | in the y-axis direction between the origins of the third coordinate system O1 and the fourth coordinate system O2 is calculated. If the condition of Dy ≧ αy occurs continuously for Ly samples, set the movement amount ye = ye1 and the movement speed yd = yd1 (after that, move to the processing when the head is directed up and down), so If not, the movement amount ye = ye2 and the movement speed yd = yd2 are set (after that, the process moves to the case where the direction is directed up and down with the eyeball).

Reference numeral 12 denotes a first determination unit that determines a right / left direction instruction from the head based on a signal from the determination unit 11 and determines whether there is an instruction from the head. That is, if the states of | xe | ≧ βx1 and | xd | ≧ βx2 occur continuously Mx times during the past Nx samples, it is determined that there is a possibility that the head has indicated the direction, otherwise It is determined that there is no instruction. Here, if it is determined that there is a possibility that the head has directed, and if the moving speed xd has changed from xd ≧ βx2 to xd ≦ −βx2, it is determined that the right has been instructed, and conversely, If the state of xd ≦ −βx2 is changed to xd ≧ βx2, it is determined that the left is instructed. If none of these apply, it is determined that there is no direction indication. FIG. 10 shows the moving state of the iris when the head rotates, and FIG. 11 shows the moving state of the iris when the eyeball rotates. FIG. 12 shows the left-right movement amount xe of the iris center, and FIG. 13 shows the left-right movement speed xd of the iris center. A similar example in which the left instruction is determined is shown in FIGS.

Reference numeral 13 denotes second determination means for determining a left / right direction instruction by the eyes (eyeball) based on a signal from the determination means 11, and determines whether or not there is an instruction by the eyeball. That is, during the past Nx samples, if the state of | xe | ≧ βx3, | xd | ≧ βx4 has occurred continuously Mx times, respectively, it is determined that there is a possibility that the direction has been indicated by the eyeball, otherwise, It is determined that there is no instruction. Here, if it is determined that there is a possibility that the direction has been indicated by the eyeball, if the moving speed xd has changed from xd ≧ βx4 to xd ≦ −βx4, it is determined that the right has been instructed, and conversely, xd If the state changes from ≦ −βx4 to xd ≧ βx4, it is determined that the left is instructed. If none of these apply, it is determined that there is no direction indication. FIG. 16 shows the left-right movement amount xe of the iris center when the eyeball is rotated, and FIG. 17 shows the left-right movement speed xd of the iris center when the eyeball is rotated. A similar example in which the left instruction is determined is shown in FIGS.

Reference numeral 14 denotes third determination means for determining an up / down direction instruction from the head based on a signal from the determination means 11 and determines whether there is an instruction from the head. That is, if the states of | ye | ≧ βy1 and | yd | ≧ βy2 occur My times consecutively during the past Ny samples, it is determined that there is a possibility that the head has indicated the direction, otherwise It is determined that there is no instruction. Here, when it is determined that there is a possibility that the head is directed, if the movement speed yd is changed from yd ≧ βy2 to yd ≦ −βy2, it is determined that the lower is instructed, and conversely, If the state is changed from yd ≦ −βy2 to yd ≧ βy2, it is determined that the above is instructed. If none of these apply, it is determined that there is no direction indication.

Reference numeral 15 denotes fourth determination means for instructing the up and down direction using the eyeball based on the signal from the determination means 11 and determines whether or not there is an instruction using the eyeball. That is, during the past Ny samples, if the states of | ye | ≧ βy3 and | yd | ≧ βy4 have occurred My times consecutively, it is determined that the direction may have been indicated by the eyeball; otherwise, It is determined that there is no instruction. Here, when it is determined that there is a possibility that the direction has been indicated by the eyeball, if the moving speed yd is changed from yd ≧ βy4 to yd ≦ −βy4, it is determined that the lower direction has been instructed, and conversely, yd If the state is changed from ≦ −βy4 to yd ≧ βy4, it is determined that the above is instructed. If none of these apply, it is determined that there is no direction indication.

  Next, the direction indicating method according to the present invention will be described in the order of steps (hereinafter, steps are indicated by S) with reference to FIG.

S1: Set the above parameters.

S2: The user 3 captures and reads the face image of the user 2 (FIG. 3) with the camera 3. The sampling frequency can be 30 Hz.

S3: Extracting the face area and setting the first coordinate system O (X axis, Y axis). That is, using an existing library such as OpenCV, a face region is extracted from a camera image, and a first coordinate system O (X axis, Y axis) with the upper left corner as the origin is set (FIG. 4).

S4: Extracting the eye area and setting the second coordinate system O ′ (X axis, Y axis). In other words, using an existing library such as OpenCV, the eye area is extracted from the face area of the camera image, and the origin coordinates (Xo axis, Yo axis) of the second coordinate system O ′ with the upper left corner as the origin are set (Fig. 5).

S5: The eye area is reset by the first low-pass filter LPF1, which is the first delay means 7, and the third coordinate system O1 (x1 axis, y1 axis) is set. That is, the first low-pass filter LPF1 is applied to the origin coordinates Xo, Yo of the second coordinate system O ′ in S2 to determine the coordinate origin (Xo1, Yo1) of the third coordinate system O1.

S6: Reset the eye area by the second low-pass filter LPF2, which is the second delay means 8, and set the fourth coordinate system O2 (x2 axis, y2 axis). That is, the coordinate origin (Xo2, Yo2) of the fourth coordinate system O2 is determined by applying the second low-pass filter LPF2 to the origin coordinates Xo, Yo of the second coordinate system O 'in S2.

In S5 and S6, the case where the first and second low-pass filters LPF1 and LPF2 are used as the first and second delay means 7 and 8 has been described. However, delaying can be performed without using such low-pass filters. You can also. That is, in S5, “N1 sample is determined by applying the first low-pass filter LPF1 to the origin coordinates Xo, Yo of the second coordinate system O ′ to determine the coordinate origin (Xo1, Yo1) of the third coordinate system O1”. Replaced the origin coordinates Xo, Yo of the previous O 'as the coordinate origin (Xo1, Yo1) of the third coordinate system O1 "and in S6," the origin coordinates Xo, Yo of the second coordinate system O' 2) Apply the low-pass filter LPF2 to determine the coordinate origin (Xo2, Yo2) of the fourth coordinate system O2 ”. The origin coordinates Xo, Yo of the second coordinate system O ′ before the N2 sample are the fourth coordinates. A method of replacing “determined as the coordinate origin (Xo2, Yo2) of the system O2” may be taken. Here, the relationship between the setting parameters N1 and N2 representing the delay amount (number of delay samples) is set as N1> N2. For example, when N1 = 5 and N2 = 0, the direction discrimination rate was about 50%.

SA: This is a step following the process of determining the left and right directions from the third coordinate system O1 by S5.
SB: This is a step following the process of determining the vertical direction from the third coordinate system O1 by S5.
SC: This is a step following the process of determining the horizontal direction from the fourth coordinate system O2 by S6.
SD: This is a step following the process of determining the vertical direction from the fourth coordinate system O2 by S6.

Next, steps following the processes SA and SC for determining the horizontal direction will be described with reference to FIG.

S7: Following the SA, the value x1c of the x1 coordinate of the iris center is obtained with respect to the third coordinate system O1 using the binarization of the image, the noise removal processing by expansion and contraction, and the labeling processing technology.

S8: Following the SC, the value x2c of the x2 coordinate of the iris center is obtained with respect to the fourth coordinate system O2 by using image binarization, noise removal processing by expansion and contraction, and labeling processing technology.

S9: S91, S92, S93 and S94.
Subsequent to S91: S7, a low cut filter is applied to the value x1c to obtain a movement amount xe1.
Subsequent to S92: S7, a differential filter is applied to the value x1c to obtain a moving speed xd1.
Subsequent to S93: S8, a low cut filter is applied to the value x2c to obtain a movement amount xe2.
S94: Following S8, a differential filter is applied to the value x2c to obtain a moving speed xd2.

S10: Based on the results of S91 to S94, a deviation Dx = | Xo1-Xo2 | in the x-axis direction between the origins of the third coordinate system O1 and the fourth coordinate system O2 is calculated.

S11: As a result of the deviation Dx calculation, it is determined whether or not the state of Dx ≧ αx occurs continuously for Lx samples.

S12: If it is determined that the state of Dx ≧ αx has occurred continuously for Lx samples, the movement amount xe = xe1 and the movement speed xd = xd1 are set. Thereafter, the process proceeds when the head is instructed to turn.

S13: If it is determined that the state of Dx ≧ αx does not occur continuously for Lx samples, set the movement amount xe = xe2 and the movement speed xd = xd2, and then proceed to the process when the direction is directed with the eyeball. .

S14: After S12, the process proceeds to the direction when the head is instructed. That is, it is determined whether or not the states of | xe | ≧ βx1 and | xd | ≧ βx2 have occurred continuously Mx times during the past Nx samples.

S15: In S14, if it is determined that the states of | xe | ≧ βx1 and | xd | ≧ βx2 have occurred continuously Mx times, the moving speed xd is further changed from xd ≧ βx2 to xd ≦ −βx2. It is determined whether or not it is turned. When it is determined that it is turned, it is determined that the instruction is right.

S16: In S15, when it is determined that the moving speed xd has changed from xd ≧ βx2 to xd ≦ −βx2, and further determined that xd ≦ −βx2 has changed to xd ≧ βx2, It is determined that the left is instructed.

In S14, if it is not determined that the states of | xe | ≧ βx1 and xd | ≧ βx2 have occurred continuously Mx times, respectively, and it is determined in S16 that the state has changed from xd ≦ −βx2 to xd ≧ βx2. If not, it is determined that there is no direction indication.

After S17: S13, the process proceeds to the process when the direction is indicated by the eyeball. That is, it is determined whether or not the states of | xe | ≧ βx3 and | xd | ≧ βx4 have occurred continuously Mx times during the past Nx samples.

S18: If it is determined in S17 that the states of | xe | ≧ βx3 and | xd | ≧ βx4 have occurred Mx times consecutively, xd further changes from the state of xd ≧ βx4 to xd ≦ -βx4 If it is determined that it is turning, it is determined that the instruction is right.

S19: When it is determined in S18 that the moving speed xd has changed from xd ≧ βx4 to xd ≦ −βx4, and it is further determined that xd ≦ −βx4 has changed to xd ≧ βx4 It is determined that the left is instructed.

In S17, during the past Nx samples, it is not determined that | xe | ≧ βx3 and | xd | ≧ βx4 are continuously generated Mx times, and in S19, xd ≦ −βx4 from the state of xd If it is not determined that ≧ βx4, it is determined that there is no direction indication.

Next, steps following the processing SB and SD for determining the vertical direction will be described with reference to FIG.

S7 ': Subsequent to SB, the y1 coordinate value y1c of the iris center is obtained with respect to the third coordinate system O1 by using image binarization, noise removal processing by expansion and contraction, and labeling processing techniques.

S8 ': After SD, the y2 coordinate value y2c of the iris center is obtained with respect to the fourth coordinate system O2 by using image binarization, noise removal processing by expansion and contraction, and labeling processing techniques.

S9 ′: S91 ′, S92 ′, S93 ′ and S94 ′.
Subsequent to S91 ′: S7 ′, a low cut filter is applied to the value y1c to obtain a movement amount ye1.
Subsequent to S92 ′: S7 ′, a differential filter is applied to the value y1c to obtain a moving speed yd1.
Subsequent to S93 ': S8', a low cut filter is applied to the value y2c to obtain a movement amount ye2.
Following S94 ′: S8 ′, a differential filter is applied to the value y2c to obtain a moving speed yd2.

S10 ': Based on the results of S91' to S94 ', a deviation Dy = | Yo1-Yo2 | in the y-axis direction between the origins of the third coordinate system O1 and the fourth coordinate system O2 is calculated.

S11 ': As a result of the deviation Dy calculation, it is determined whether or not the state of Dy ≧ αy has occurred continuously for Ly samples.

If it is determined that the state of S12 ′: Dy ≧ αy has occurred continuously for Lx samples, the movement amount ye = ye1 and the movement speed yd = yd1 are set. Thereafter, the process proceeds when the head is instructed to turn.

S13 ': If it is determined that the state of Dy ≧ αy does not occur continuously for Ly samples, set the movement amount ye = ye2 and the movement speed yd = yd2, and then the process when the direction is indicated with the eyeball Move.

S14 ': Following S12', the process proceeds to the direction when the head is instructed. That is, it is determined whether or not the states of | ye | ≧ βy1 and | yd | ≧ βy2 have occurred continuously My times during the past Ny samples.

In S15 ': S14', if it is determined that the states of | ye | ≧ βy1 and | yd | ≧ βy2 have occurred continuously My times, the movement speed yd is further increased from the state of yd ≧ βy2 to yd ≦ − It is determined whether or not it has changed to βy2, and when it is determined that it has changed, it is determined that the instruction is down.

S16 ′: In S15 ′, when it is determined that the moving speed yd has not changed from xd ≦ −βy2 to yd ≧ −βy2, it is further determined that yd ≦ −βx2 has changed to yd ≧ βy2. If it is, it is determined that the top is instructed.

In S14 ′, if it is not determined that the states | ye | ≧ βy1 and | yd | ≧ βy2 have occurred continuously Mx times, and in S16 ′, the state yd ≦ −βy2 is changed to yd ≧ βy2. In the case where it is not determined that there is any direction, it is determined that there is no direction instruction.

Following S17 ': S13', the process proceeds to the case where the direction is indicated by the eyeball. That is, it is determined whether or not | ye | ≧ βy3 and | yd | ≧ βy4 have occurred continuously My times during the past Ny samples.

S18 ': In S17', if it is determined that the states of | ye | ≧ βy3 and | xd | ≧ βy4 have occurred continuously My times, the movement speed yd is the movement speed from the state of yd ≧ βy4. It is determined whether or not yd ≦ −βy4 has been reached.

S19 ′: In S18 ′, when it is not determined that the moving speed yd is changed from yd ≧ βy4 to yd ≦ −βy4, it is further determined that yd ≦ −βy4 is changed to yd ≧ βy4. If it is, it is determined that the top is instructed.

In S17 ', if it is not determined that the states of | ye | ≧ βy3 and | yd | ≧ βx4 have occurred continuously My times during the past Ny samples, and the state of yd ≦ -βy4 in S19' If it is not determined that yd ≧ βy4, the direction is not determined.

Next, the verification result of the determination accuracy is shown.
a. Subjects: 3
b. Contents of verification: A total of 40 trials were performed, in which the direction of the eye was directed 10 times to the left and right, and the direction of the head was directed 10 times to the left and right.
c. Verification method: (1) When the first and second low-puff filters LPF1 and LPS2 are used as the first and second delay means (the method of the present invention) and (2) When the conventional method does not use the low-pass filter The direction indication discrimination rate was compared.
d. Verification result: Direction discrimination rate of 3 people (average value)
(1) Method of the present invention: eye movement 81%, head rotation 98%
(2) Conventional method: eye movement 81%, head rotation 0%
As a result of the verification, it was found that in the eye rotation movement, the discrimination rate almost the same as that of the conventional method was obtained in the present invention, and the direction discrimination ratio according to the present invention was much superior in the head rotation motion.

In the verification experiment, we intentionally selected one person with large eyes (a person in a normal condition with wide eyes) and one with narrow eyes (a person in a normal condition with little eyes open). When the eye opening degree A = (eye height) / (eye length) is used as an index representing the size of the eyes, the following is obtained.
・ Subject 1: A = 0.39 (person with large eyes)
・ Subject 2: A = 0.34
・ Subject 3: A = 0.28 (thin with narrow eyes)
As a result of experiments, it was found that those with larger eyes can obtain a higher discrimination rate. Subject 1 had a discrimination rate of 100% eye movement and 100% head rotation, while subject 3 had a discrimination rate of 55% eye movement and 95% head rotation.

1 Computer 2 User (operator)
3 Camera (imaging means)
4 mark
5 First coordinate system setting means 6 Second coordinate system setting means 7 First delay means 8 Second delay means 9 First calculation means 10 Second calculation means 11 Determination means 12 First determination means 13 Second determination means 14 Third Determination means 15 Fourth determination means

Claims (4)

Imaging means for imaging a computer operator's face area;
First coordinate system setting means for setting a first coordinate system of the face area imaged by the imaging means;
Second coordinate system setting means for setting a second coordinate system of the eye area in the face area imaged by the imaging means;
First delay means for obtaining a third coordinate system that moves with a delay following the movement of the eye area;
Second delay means for obtaining a fourth coordinate system that moves with a delay following the movement of the eye area and has a delay time smaller than that of the first delay means;
First calculating means for calculating the moving amount and moving speed of the iris center in the third coordinate system;
Second calculating means for calculating the moving amount and moving speed of the iris center in the fourth coordinate system;
Discriminating means for discriminating operation based on the distance between the origins of the third coordinate system and the fourth coordinate system;
First determination means for determining a left-right direction instruction from the head based on a signal from the determination means;
Second determination means for determining a left-right direction instruction by an eyeball based on a signal from the determination means;
Third determination means for determining a vertical direction instruction by the head based on a signal from the determination means;
Fourth determination means for determining a vertical direction instruction by an eyeball based on a signal from the determination means;
Direction indicating device characterized by comprising The first delay means comprises a first low-pass filter that performs low-pass filtering on the origin coordinates of the second coordinate system, and the second delay means performs low-pass filtering on the origin coordinates of the second coordinate system. 2. The direction indicating device according to claim 1, comprising a second low-pass filter having a delay time smaller than that of the one low-pass filter. Imaging the face area of the computer operator;
Setting a first coordinate system of the face area imaged by the step;
Setting a second coordinate system of the eye area image in the captured face area;
Acquiring a third coordinate system that follows and moves with the movement of the eye region with a delay;
Accompanied by the movement of the eye region, the fourth coordinate system that follows and moves with a delay and has a smaller delay time than the third coordinate system;
Calculating a moving amount and moving speed of the iris center in the third coordinate system;
Calculating a moving amount and moving speed of the iris center in the fourth coordinate system;
Performing an operation determination based on a distance between origins of the third coordinate system and the fourth coordinate system;
Determining left and right direction instructions from the head based on the motion determination;
Determining a left-right direction instruction by an eyeball based on the operation determination;
Determining a vertical direction instruction by the head based on the motion determination;
Determining a vertical direction instruction by an eyeball based on the motion determination;
Direction indication method characterized by comprising The step of obtaining the third coordinate system is performed by a first low-pass filter that performs low-pass filter processing on the origin coordinates of the second coordinate system, and the step of obtaining the fourth coordinate system comprises obtaining the origin coordinates of the second coordinate system. 4. The direction indication method according to claim 3, wherein a low-pass filter process is performed by a second low-pass filter having a delay time smaller than that of the first low-pass filter.

Images