JP2003150306A - Information display device and method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a user-friendly user interface by which information, merchandise, etc., are easily selected by using visual lines of a user, prompt responses are enabled in every device and which does not hinder natural actions of the user. SOLUTION: The user performs selection by the visual lines, a gazed point is estimated from the visual line direction by a gazed point estimation part 30, a desired pieces of information, merchandise, etc., are predicted by a prediction part 40 and pieces of the information, merchandise, etc., to be selected objects are displayed by an information presentation part 20. Consequently, a burden of selection for the user is relieved.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a face-to-face device system such as a vending machine or a ticket vending machine, an OA device in a personal computer or a workstation, and an information providing device such as a tourist guide. The present invention relates to an information display device using a location estimation device and a method thereof.

[0002]

2. Description of the Related Art In a face-to-face device system such as a vending machine or a ticket vending machine, an OA device such as a personal computer or a workstation, an information providing device such as a tourist guide, a product, a ticket, or information desired by a user Etc. are transmitted to the system, input with buttons, keyboard, mouse,
In addition, there is an input method for selecting what is displayed on the screen using a touch panel.

However, in a situation where it cannot be operated by hand immediately,
In addition, there are situations where you can fully instruct without using hands,
In this case, in order to input the user's intention, the above-mentioned input device impedes human actions and compels a troublesome operation.

Particularly in recent devices, a large number of buttons are arranged, and when selecting desired information or product, "to grasp the configuration of the entire system → search while looking at the desired one → hold a hand at the place" Since there is a case where it takes a lot of time to make one selection due to the flow of actions such as “going”, it is considered that another input means is necessary for the system that requires prompt response.

As a method of selecting a position without using a hand, there is a line-of-sight detection. The information obtained from the line of sight includes not only position information but also human intentions, intentions, and the like, which facilitates the task of selecting an interesting one from a plurality of objects.

Conventionally, there have been many inventions for detecting the line of sight, and there are the following two types.

The first is a contact type using an instrument such as glasses represented by an eye camera, and the second is a CCD.
It is a non-contact type in which the line of sight of a person is detected from an image taken by using a camera or the like.

In the contact type, a device such as spectacles and goggles is attached, and a line of sight is input by a detection device attached to the device.

In the non-contact type, a face is photographed from a distant position with a CCD camera, and the line-of-sight direction is estimated using the feature amount near the eyes.

For example, in Japanese Unexamined Patent Publication No. 4-255015,
The relative movement amount of the line of sight is obtained from the change in the center coordinates of the extracted pupil contour data. Although the present invention has an error accumulation and a mechanism for compensating for it, it is insufficient.

Further, in Japanese Patent Application Laid-Open No. 3-17696, for the purpose of controlling the cursor position of a word processor or the like by the line of sight, the eye region is divided by a vertical line and a horizontal line passing through the center of the pupil, and the divided region is divided. The viewpoint position is calculated using the area ratio as a feature amount.

Generally, as a problem of the visual axis detection, in order to improve the accuracy of the visual axis detection, a physical restriction such as a contact type or fixing the head is required. There is a problem that it is difficult to use an input device that interferes with human's natural behavior or that requires the user to wear the device.

As an input device using a line of sight, Japanese Patent Laid-Open No.
In 372012, there is a method in which voice recognition is combined with line-of-sight detection, motion detection such as blinking, and the like is complementarily used to communicate an intention and input information.

In addition, the literature (Integrating Simultaneous I
nput from Speech, Gazo, and handGestures David BK
oons, Carlton J. Spartell, and Kristinn R. Thorisso
n, Intellelligent Multimedia Interfaces Edited by M
ark T. Maybury, MIT Press (1993)) describes how to integrate gaze, speech recognition, and hand gesture-based interfaces. This prototype system exchanges map information and operates in a three-dimensional block world.

Most of these inventions using the line of sight are mainly used as a substitute for a keyboard or a mouse, and the accuracy of the line of sight detection remains a problem.

[0016]

By the way, as an input device using the line of sight, there is JP-A-4-372012.
This is because the presentation of information is limited to the display, and it is intended to identify an icon or the like displayed on the display with a line of sight and operate it by a voice instruction means. However, in order to perform these operations, a high-precision line-of-sight detection means is required, and there are many difficult parts to substitute for the current interface of a mouse or the like.

In view of the above-mentioned problems, the information selected by the user is efficiently presented in consideration of the accuracy of line-of-sight detection, and the information and the like can be easily provided without requiring more than necessary accuracy for line-of-sight detection. The purpose is to provide a device that can be selected. For example, even if a line-of-sight detection means capable of distinguishing the degree to which the display is divided into four parts is prepared, the information can be selected by efficiently presenting the information.

Further, the present invention can provide an interface which allows "searching for and selecting desired information" by using the line of sight and other motions at the same time and allowing the user to take natural actions. For example, when the user selects the desired product, information, place, etc. with his line of sight, presses one of the installed buttons, or utters a set word, The purpose is to determine the point of gaze using the result of the estimated direction of and to enable quick selection and designation of information.

Further, by predicting human behavior in advance and presenting the information that is considered most suitable when the system presents the information, it is possible to reduce the burden of information selection on the user. In addition, when inputting, the so-called "timing" that dynamically changes at what time information is selected
The purpose of this is to provide a device that is easier to use from a human standpoint by controlling the.

[0020]

According to a first aspect of the present invention, there is provided an information display device for a user to select necessary information from a plurality of pieces of information. A viewpoint estimating means, a predicting means for predicting information selected by the user from the gazing point estimated by the gazing point estimating means, and an information display means for displaying the information predicted by the predicting means. This is an information display device characterized by the above.

As a result, the user makes a selection based on the line of sight, the gazing point is estimated from the direction of the line of sight by the gazing point estimating means, the predicting means estimates the desired information, the product, etc., and the information display means selects the selection target. Display information, products, etc. Therefore, the burden of selection by the user is reduced.

The predicting means in the invention of claim 2 is
When the gaze point estimated by the gaze point estimating means moves, a prediction changing means for re-estimating the information selected by the user, and the information re-estimated by the prediction changing means are displayed. The information display device according to claim 1, further comprising a display changing unit that instructs the information displaying unit.

Thus, the display changing means changes the display portion of the information at the gazing point so that the user can perceive which information is selected by the line of sight.

The predicting means in the invention of claim 3 is
When the gazing point estimated by the gazing point estimating means is in the information displayed on the information display means by the user, the information selected by the user based on the gazing information is displayed. 2. The information display device according to claim 1, wherein when the gazing point is other than the information displaying means, the information selected by the user is predicted based on the place where the gazing point is. Is.

Thus, the display content can be changed from the action of looking at a place other than the information display means.

The invention according to claim 4 is an information selecting method for a user to select necessary information from a plurality of information, and a gaze point estimating step of estimating a gaze point of the user, A prediction step of predicting the information selected by the user from the fixation point estimated by the fixation point estimation step; and an information display step of displaying the information predicted by the prediction step on the information display means. This is a characteristic information selection method.

The prediction step in the invention of claim 5 further comprises: a prediction changing step for re-predicting the information selected by the user when the gazing point estimated by the gazing point estimating step moves, 5. The information selecting method according to claim 4, further comprising a display changing step for instructing the information display means to display the information re-estimated by the prediction changing step.

In the prediction step in the invention of claim 6, the gazing point estimated by the gazing point estimating step is
When the user is in the information displayed on the information display means, the information selected by the user is predicted based on the information being watched, and the gazing point is other than the information display means. In the case of (1), the information selection method according to claim 4, wherein the information selected by the user is predicted based on the place where the user is gazing.

[0029]

BEST MODE FOR CARRYING OUT THE INVENTION An information selection device 10 according to an embodiment of the present invention will be described below. In this embodiment, a case where the information selection device 10 is incorporated in an automatic vending machine, a ticket vending machine, or the like will be described. In addition, the "system" in the description is a vending machine,
It means a device such as a ticket vending machine.

A block diagram of the information selection device 10 is shown in FIG.

The information selection device 10 includes a display and a CR.
Information presentation unit 20 such as T, liquid crystal, panel, signboard, display plate, etc.
A gazing point estimation unit 30 that estimates the gazing point of the user, a prediction unit 40 that predicts the selection contents of the user, a display changing unit 50 that changes the display contents, and a signal from the user or the system. It includes an intention transmitting unit 60 for inputting and a result output unit 70 for holding the selection result.

[1] Information presenting unit 20 The information presenting unit 20 includes an information displaying unit 22 such as a display,
The information storage unit 24 and the display drive unit 26 are shown in FIG.

Information such as images and character strings registered in the information storage section 24 is stored in combination with the display position information, and the display drive section 26 displays the information on the information display section 22 in accordance with the display position information. Do.

The information storage unit 24 and the display drive unit 26 are implemented by using a general-purpose personal computer or the like according to the flowchart shown in FIG.

The display drive section 26 changes the display contents according to an instruction from the display change section 50 described later. The "state" in the state determination determines a state such as an initial state, some states in the predicted state, and a confirmed state after the intention input is performed. In each state, the information stored in the information storage unit 24 is presented.

When the system is a vending machine, products A to D
Is displayed as shown in FIG.

The information and the like to be selected are displayed relatively large in the information presenting section 20. Regarding the presentation of information,
The display is performed in a size considering the spatial resolution of the gazing point estimation unit 30. In the case of the present embodiment, the screen of the display is divided into four parts, and the line-of-sight detection means having a low spatial resolution can be used for sufficient discrimination.

[2] Point-of-regard estimation section 30 Normally, when a person searches for a button or display section to be selected, he or she looks at the place where the target is displayed and installed. The gaze point estimation unit 30 that acquires the gaze direction and estimates the gaze point will be described (see FIG. 5).

The gazing point estimating section 30 includes an image input section 31, a face direction detecting section 32, a pupil position detecting section 33, and a gazing point determining section 3.
4 and a database unit 35.

[2-1] Image Input Unit 31 The image input unit 31 acquires a monocular color image, and the obtained image is sent to the face direction detection unit 32.

[2-2] Face Direction Detection Unit 32 The face direction detection unit 32 for determining which direction the face is facing will be described.

(1) The face direction detection section 32 cuts out a face as shown in FIG. 6 (a).

In this method, the acquired image is divided into areas, and the face area is found from the hue and saturation information of each area.

(2) Next, as shown in FIG. 6 (b), extraction of face parts such as cutting out of the eyes and mouth is performed with reference to FIG.

By binarizing the hue image for the face area portion, candidates for an eye portion and a mouth portion having different color attributes from the skin color portion are obtained. Next, a region generated by noise such as a shadow is selected according to the shape constraint. The feature quantity expressing the region shape is quantified using a moment invariant or an autoregressive model. Further, the selected area is compared with the light and shade pattern of the eyes and mouth to determine the eye area and the mouth area. Matching with the gray pattern here means
A dictionary for recognition is preliminarily constructed from dyed partial images of the eyes and partial images of the mouth by KL expansion, and a pattern similar to each pattern is selected using the composite similarity or the like.

(3) After detecting the eyes and mouth, edge images are generated for the right eye, the left eye, and the mouth as shown in FIG. 6 (c). A conventional Sobel filter or Canny method is used to generate the edge image.

(4) Then, the edge image is mapped perpendicularly to the one-dimensional axial direction to detect the inner corner of the eye. The same process is performed for the mouth portion to detect the left and right mouth areas. The image is scanned in the direction perpendicular to the one-dimensional axis, and the point having the lowest lightness is set as the inner corner point.

(5) As shown in FIG. 6 (e), the method of calculating the normal direction of the face surface by using the distortion target by the distortion of the plane composed of four points (AHGee and R. Cipolla: Deterr
niningthe Gaze of Faces in Images, CUED / F-INFENG / TR
-174 (1994)), but here, it is obtained by performing linear interpolation from the information on the positional relationship of the four points and the information on the face direction which have been obtained in advance by experiments, and creating a database.

(7) For example, for the positions of four points shown in FIG. 6 (f), the parameter in the horizontal direction is α and the parameter in the vertical direction is β by the following equation.

[0050]

[Equation 1] The expressions α and β are not limited to this, and other facial features may be used when the orientation of the face is determined.

[2-3] Pupil Position Detecting Unit 33 Next, the pupil position detecting unit 33 for determining the position of the pupil in the eye will be described.

(1) From the result of cutting out the eyes obtained by the face direction detecting unit 32, the partial image is binarized as shown in FIG. 8A. The center of gravity of the black pixel is obtained from the binarization result, and the pupil portion is detected.

(2) The center of gravity of the pupil is obtained, and the vector V is obtained from the previously obtained inner canthus point (FIG. 8 (b)).

(3) On the other hand, two straight lines A'and A "obtained by the face direction detecting section 32 and passing through the inner and inner corners of the eye.
Then, the slopes of the two straight lines are averaged to obtain the straight line A (FIG. 8C).

(4) The angle between the vector V and the straight line A is θ,
Let σ be the length of the vector V. These θ and σ are used as parameters of the position of the pupil.

[2-4] Point-of-regard determining section 34 The point-of-regard determining section 34 determines which location of the information presenting section 20 is viewed with respect to the line-of-sight direction. Face orientation α, β
The gazing point is obtained from the eye positions θ and σ at.

FIG. 9 shows a face orientation (f (α)) and eye positions (g (θ,
σ)). here,

[Equation 2] And a and b are parameters considering individual differences and the like. The functions of f and g are not limited to this, and may be changed depending on the accuracy to be obtained, individual differences, and the like.

The information presentation unit 2 is shown in FIG.
It is 0 and is divided into four in the horizontal direction. At this time, as shown in FIG. 9B showing the relationship between the face direction and the position of the eyes, the subspace of the parameters is obtained when looking at the divided places. Qualitatively speaking, it represents a relationship such as "even if the face is facing to the right, the eyes are looking to the left if they are facing to the left".

The same can be determined for the vertical direction.

[2-5] Database unit 35 In the database unit 35, several kinds of data shown in the lower part of FIG. 9B are prepared according to the distance from the system. When the distance is far, the gaze direction can be determined by emphasizing the orientation of the face.

The gazing point estimation unit 30 sends information as a set of (distance, gazing direction) to the prediction unit 40 so that the gazing direction according to the distance can be determined.

[3] Prediction unit 40 The prediction unit 40 predicts information selected by the user.

The predicting section 40 comprises a distance estimating section 41, a database section 42, an information recording section 43 and a predictive information composing section 44, and a block diagram is shown in FIG.

[3-1] Distance Estimating Unit 41 The distance estimating unit 41 calculates the distance between the user and the system. In this embodiment, observation with a stereo camera is performed and the distance to the user is obtained by triangulation. In addition,
Other measurement methods using an ultrasonic sensor or the like may be used, or the distance may be obtained from the area of a person photographed by a monocular camera.

The distance is divided into about 3 to 5 and the prediction content at each distance is changed. In this embodiment, it is divided into three, and the system operable range (tens of centimeters) from a short distance,
System visible range (several meters) and system neglected range (three levels or more).

[3-2] Database Unit 42 The database unit 42 has two presentation content databases and action databases.

The presentation contents database holds data in which the position information and contents in the information presenting unit 20 installed in the system are linked. The position information is represented by a (x1, y1)-(x2, y2) rectangle in which the information is located, and the content and the display form on the information presenting unit 20 are represented as a set.

In the behavior database, some patterns are prepared in advance for the behavior patterns of the user. Especially, it is used to prepare a behavior pattern when approaching the system while operating the system and to know what behavior pattern is close to.

[3-3] Information Recording Unit 43 The information recording unit 43 records the line-of-sight direction at regular intervals.

The line-of-sight direction is selected and determined from the information sent from the gazing point estimation unit 30 according to the distance obtained by the distance estimation unit 41, and the line-of-sight direction is recorded.

The information recording section 43 is constructed by using a ring buffer.

The ring buffer is schematically represented as in the lower part of FIG. For example, if information is stored in the order of 1 to 8, then 1 is deleted and new information is added. Since the rewriting is sequentially performed in this manner, the information before the fixed time is automatically updated. The recorded contents are registered in the ring buffer as a set of the line-of-sight direction obtained from the gazing point detection means, the distance information obtained from the distance detection unit, and the acquisition time (line-of-sight direction, acquisition time, distance) (FIG. 12
See middle section).

[3-4] Prediction Information Composition Unit 44 The prediction information composition unit 44 collates the content recorded in the information recording unit 43 with the action pattern information stored in the database unit 42 to determine what information, Predict whether you are looking for products etc.

The action pattern information is time-series information,
It is represented as in FIG. Note that the gazing point in the figure is actually a two-dimensional movement of the viewpoint, but it is expressed one-dimensionally for the sake of explanation.

FIG. 11 (b-1) shows that "approaching while gazing at a certain point".

FIG. 11B-2 shows that "a fixed point is being observed while approaching to a certain point, and then the system is approached and approaching".

FIG. 11 (b-3) shows that "approaching slowly".

FIG. 11 (c) shows that "you are moving far away".

Some of these behavior patterns are prepared and registered in the database unit 42.

The prediction method changes the function in consideration of the distance to the user.

[3-5] System Operable Range As an action pattern in the system operable range, (a-
1) "Continue to see what you are interested in" (a-2) "Get closer to what you are interested in" (a-3) "I am not interested and want other information" (a-4) "I'm not interested" etc.

(A-1) "Keep seeing what you are interested in"
(A-2) In the case of “Look closer to something of interest” or the like, information about the point of interest regarding which information is selected is sent to the result output unit 70.

When information that is not of interest is presented, or when the gazing point is directed to the outside of the information presenting section 20, it is determined that he is not interested, as in (a-3). Determines that new information is needed. Regarding this, the function will be described later in the display changing unit 50.

[3-6] System visible range In the system visible range, the user is not only the system,
It is possible to see other informational presentations. For example, when purchasing a ticket for a Shinkansen at a ticket vending machine, a route map or the like is usually displayed above the ticket vending machine line, and the destination and the fare are confirmed by looking at the route map. At this time, if you can obtain information about where you were looking on the route map, by displaying some toll sections that you will purchase, you can quickly select the target ticket without displaying the unnecessary sections It will be possible.

Consider this situation as the case (b-1) of the action pattern listed above in FIG. At this time, it is possible to determine what information to present based on what the previously viewed information is. For example, in the example above, depending on whether you were looking at the east side or the west side of the route map,
For example, some candidates of the desired place are presented. At that time, the information in the ring buffer is as shown in FIG.
If it is in the middle row, the content of information presentation is determined based on the information at the gazing point (x1, y1) seen at a long distance. By presenting expected information in this way, the operation time can be shortened.

[3-7] System neglected range In the system neglected range, if the behavior pattern shown in FIG. 11 (c) is met, no prediction is made and the initial state remains.

In the case where the action pattern is as described in each figure, when the one-dimensional waveform pattern is used, the collation method is as follows:
The sum of squared differences is set as the similarity, and the one having the highest similarity is selected as the action pattern. As a matching method,
A conventional pattern recognition method such as simple similarity or composite similarity, DP matching, a neural network, or the like may be used. If the action pattern does not apply or if the processing fails, the correspondence with the human being is started from the set initial state.

When the instruction content is changed by an instruction from the prediction section 40, there are a method of displaying the line-of-sight position and a method of changing the display content of the information to be selected. The prediction unit 40 sends an instruction about the line-of-sight position and the content of the prediction to the display changing unit 50. When changing the content of the presentation, the information presenting unit 2
When 0 is instructed to change the state, the information presenting unit 20 determines the information presenting unit 2 based on the information obtained by the estimating unit 40.
0 determines the state and presents information.

Further, the predicted action pattern and the selection contents at each time are sent to the result output unit 70. At this time, not only one piece of the selected information is sent, but all the selection contents of the time when the system is in the exchange for making the selection are sent to the result output unit 70. What is selected as one is determined as the selection content that is being viewed at the time designated by the result output unit 70.

[4] Display changing unit 50 The display changing unit 50 comprises a gazing point display unit 51 and a selection information display changing unit 52, and is shown in FIG.

[4-1] Point-of-regard display section 51 The point-of-regard display section 51 displays the information portion at the position of the point-of-regard sent from the prediction section 40, for example, by changing the color.
Alternatively, the display such as inversion is changed so that the user can perceive what information the user is currently gazing at. For example, as shown in FIG. 14, when the product B is viewed (a) and when the product D is viewed (b), it is inverted and output. When it is confirmed, the display is changed as shown in (c).

[4-2] Selection information display changing unit 52 The selection information display changing unit 52 changes the presentation content based on the instruction sent from the prediction unit 40. A method for realizing a function of scrolling the display information in the direction when the user's point of gaze comes out of the information presenting unit 20 will be described.

Generally, when the number of information that can be displayed is small,
For example, when only four different pieces of information can be displayed as shown in FIG. 4, and when it is desired to select as many as 24 pieces of information as shown in FIG. 15, normal hierarchical selection is performed. As shown in the column on the left side of FIG. 15, first select from the large classifications on the upper left,
Next, the middle classification on the left middle row is selected, and finally the desired one (here, D is selected) is hierarchically selected. However, as an advantage of using the line of sight, it is possible to show the intention "I do not see if the information is not displayed", and it is possible to look in the direction where the desired information is likely to be. The information display method as shown in the right column of FIG. 15 is possible.

On the right side of FIG. 15, a region surrounded by a thick line shows four pieces of information displayed by the information presenting section 20. Now, look at the direction of the place where the target information (D in the example) is located. That is, as shown in the middle right, the information presenting unit 2
By viewing the X mark outside 0, the displayed place is changed, and the information presentation is changed as shown in the lower right.

For the coordinate system (x, y) in FIG. 15, the range of the screen is (0, 0)-(xs, ys).
If the point of interest is outside the range (0,0)-(xs, ys) of the screen, such as (xp, yp), if there is information in that direction as the content of the presentation information, the presentation information To change. FIG. 17 is a flowchart thereof.

At this time, the line of sight of the user is actually outside the information presenting section 20, but it is sufficient for a human to confirm whether or not the presented information has changed by peripheral vision.
When the display is changed or when the user directs his or her line of sight to the inside of the information presenting unit 20, the point of interest is displayed again.

Further, in the case of using the visual axis detection device having a low spatial resolution, the following function for changing the display resolution is realized when a small display is being watched. As shown in FIG. 16, when a detailed display is performed, when the elliptical portion in the drawing is continuously watched, and when display information below a certain level is displayed with a high resolution, Display information is also stored, and the display content of the gaze portion is displayed in the information presentation unit 20 as shown in the center of FIG.
It will be displayed larger. Further, as shown in the lower part of FIG. 16, it is possible to change the size hierarchically. By changing the display resolution using such line-of-sight information, it is possible to present information that is easier to select.

[5] Intention transmission unit 60 The intention transmission unit 60 includes an input unit 61 for inputting signals such as voice and switch-on / off, and an input determination unit 6 for recognizing the contents of the input.
2 (see FIG. 18).

The input section 61 is a microphone, a switch, a mouse, a camera or the like, and sends the input result to the input determination section 62. The input determining unit 62 considers the type and content of the input device, analyzes the input content, and sends the input information to the result output unit 70.

Input judging unit 6 when voice is used as an input device
As a block diagram of No. 2, a voice recognition device is used for the input determination unit 62. For example, by uttering “Please give me” while looking at a certain product, voice is input from the input device to the voice recognition device, and the recognition result is sent to the result output unit 70.

By outputting the information of the gazing point at the same time when the input information is input, the information of the place where the user is looking is selected. For example, by uttering a simple word such as “please do” or “this” while the user looks at the desired information, the information of the point of gaze at the time of uttering can be selected.

As another input example, in the embodiment of the vending machine, the input section 6 attached to the insertion slot for coins, bills, etc.
The input signal may be that money is thrown in by 1.

[6] Result Output Unit 70 The result output unit 70 includes the gaze information holding unit 71 and the state holding unit 7.
2. The timing adjustment unit 73 is provided, and its configuration values are shown in FIG.

[6-1] Gaze information holding unit 71 The gaze information holding unit 71 holds the information gazed by the user, the product, and the like. The information predicted by the prediction unit 40 is also held.

[6-2] Timing Adjusting Unit 73 The timing adjusting unit 73 adjusts the simultaneity between the time when the presented one is selected by the line of sight and the time input by the input unit 61. When input information is input, set when to output the gaze point information. The timing adjustment unit 73 is a unit that evaluates the input time of one or a plurality of input units 61 and determines the time of intention transmission, and includes a timing setting unit 74 and a time setting unit 75.

The operation of the timing adjusting section 73 is shown in FIG.

In the timing setting section 74, the timing for each action pattern is found by experiments or the like, and the timing is set so that the user can easily use it. The time setting unit 75 sends to the result output unit 70 the total time of the time set by the timing setting unit 74 and the time when the signal is received from the input determination unit 62.

For example, when the time tk = 0 set by the timing setting unit 74 is set, (a-1) when the “keep watching” pattern is performed, the time tin input from the input judging unit 62 is The time made is sent to the state holding unit 72.

As another input example, in the embodiment of the vending machine, the input section 6 attached to the insertion slot for coins, bills, etc.
The case where the input signal is input of money according to 1 will be described. In this case, the input determination unit 62 determines the amount of money input. The timing setting means selects a product having a gazing point at a time after the timing setting time tk from the time t1 when the money is inserted. tk is -1.0
By setting (sec) ≤ tk ≤ 1.0 (sec), "insert coins while looking at product" allows the product to be selected without the need to press a button or the like. In this case, it is considered that the user's line of sight is often at the money insertion port when the money is inserted. Therefore, it is possible to select the product by setting the timing setting time tk after a few seconds and gazing at the product for a while after the money is inserted.

[6-3] State Holding Unit 72 The state holding unit 72 outputs a state such as an initial state, a predicted state and a fixed state, holds the state and the gaze information in the state, and the system or other means. Send to. In the confirmed state, the system receives information on the user's point of gaze, information on products, etc. from the result output unit 70, and performs sales, start of services, and the like. Further, another operation such as displaying information may be performed.

(Modification) A modification will be described.

[1] Modification of Information Presenting Section 20 The information presenting section 20 is not limited to the dynamically changing one as displayed on the display. For example, a fixed product, an information panel, or the like may be installed in or near the device.

When the information presenting section 20 is not dynamically displayed on the display but fixed information is presented, even if a lamp, LED or the like is installed so that the direction of the gazing point can be known. good.

[2] Modified Example of Gaze Point Estimating Unit 30 In the embodiment, the non-contact line-of-sight detection method using a monocular camera has been described. However, the line-of-sight detection method is not limited to contact or non-contact.

For example, a document (Akira Banno, Fumio Kishino, Yukio Kobayashi “Trial manufacture of eye-gaze detecting device which allows pupil extraction processing and head movement”), IEICE Transactions (D) Vol.J76-D-II , No.
3, pp.636-646 (1993)), a method in which a person is photographed by using a plurality of cameras and the face orientation and the eye orientation are obtained may be used.

An accurate line-of-sight detection method can be applied not only to devices intended for short-term correspondence such as the case of an automatic vending machine, but also to an application example in which a large amount of data or the like needs to be further divided. May be required. In this case, a contact type visual line detection method such as an eye camera or an eye tracker may be used.

[3] Modification of Intention Transmitting Unit 60 The intention transmitting unit 60 is not limited to a voice or an input method such as the insertion of money as shown in the embodiment, but a means for attaching a button or the like to the system, The following transmission methods can be given because the orientation of the face is detected. Move your neck up and down with the same point of view ("nodling" movement)
And so on. In this way, the operation detecting means based on the image information may be added to change the input.

Further, in the case where the intention transmitting unit 60 is provided with a high-level input judging unit such as voice recognition, the input contents may be changed to a configuration for explicitly changing the display contents. Specifically, by inputting "next content" or the like by voice, the display changing unit 50 presents another information.

Instead of the mouth detecting means, a nostril detecting means for detecting a nostril may be used. As the structure of this nostril detecting means, since binarization of the hue image has already found a portion of a color different from the skin-colored portion when the candidates for eyes and mouth are obtained, two nostrils are provided for each portion. The pattern is collated with a dictionary generated from the partial image of the nostrils photographed so as to include two nostrils at the same time.

Then, in the case of detecting the face direction, the surface may be constructed so as to connect the two inner canthus and the two nostrils (see (g) of FIG. 6).

[4] Modification of Display Change Unit 50 In the gazing point display unit 51, not only the color conversion inversion of information but also the
The point of interest may be displayed as it is with a cursor.

Further, the gazing point display section 51 itself may be removed. In other words, the purpose is to enable humans to constantly perceive that the gazing point is changing,
There may be applications where you don't want it to be perceived. In that case, the intention transmitting unit 60 may be modified so that the gazing point can be confirmed only when specified.

The selection information display changing section 52 may be combined with the intention transmitting section 60 to scroll the information only when an intention is transmitted, for example, a signal "I want you to change the display" is sent.

Further, the display changing section 50 may not be provided, or a method of making a prediction based on a fixed display content and making a selection may be used.

As described above, the present invention can be variously modified and implemented without departing from the spirit thereof.

[0126]

According to the present invention, it is possible to easily select information, merchandise, and the like using the line of sight of the user, and it is possible to quickly respond to any device. In addition, a user-friendly user interface can be provided without disturbing human's natural behavior.

[Brief description of drawings]

FIG. 1 is a block diagram of an apparatus.

FIG. 2 is a block diagram of an information presenting unit 20.

FIG. 3 is a flowchart of an information presenting unit 20.

FIG. 4 is a diagram of an example of information presentation.

5 is a block diagram of a gazing point estimation unit 30. FIG.

FIG. 6 is an explanatory diagram of a face direction detection unit 32.

FIG. 7 is a diagram of extraction of face parts.

FIG. 8 is an explanatory diagram of a pupil position detection unit 33.

FIG. 9 is an explanatory diagram of a method of determining a gazing point.

FIG. 10 is a block diagram of a prediction unit 40.

FIG. 11 is a diagram of an example of an action pattern.

FIG. 12 is a diagram of an example of information storage of an information recording unit 43.

FIG. 13 is a block diagram of a display changing unit 50.

FIG. 14 is a diagram of a display change example in the information presenting unit 20.

FIG. 15 is a diagram illustrating a scroll function example in a display information change example.

FIG. 16 is a diagram of an example of resolution conversion in an example of changing display information.

FIG. 17 is a diagram of an operation example of a selection information display changing unit 52.

FIG. 18 is an intent. 6 is a block diagram of a transmission unit 60. FIG.

FIG. 19 is a flowchart of a timing adjusting section 73.

20 is a block diagram of a result output unit 70. FIG.

FIG. 21 is a block diagram of a timing adjusting unit 73.

[Explanation of symbols]

10 Information selection device 20 Information presentation section 30 Gaze point estimation section 40 Predictor 50 Display change section 60 Intention transmission part 70 Result output section

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Kazuhiro Fukui             1-30 Oyodochu, Kita-ku, Osaka City, Osaka Prefecture             Umeda Sky Building Tower West Stock Association             Company Toshiba Kansai branch office F term (reference) 5B087 AA09 AB09 BC05 BC12 BC13                       BC26 BC32 DE03

Claims (6)

[Claims] 1. An information display device for displaying information, comprising: a gaze point estimating means for estimating a gaze point of a user who uses the information display device; and a gaze point estimated by the gaze point estimating means. An information display device comprising: a prediction unit that predicts information required by the user, and an information display unit that displays the information predicted by the prediction unit. 2. The predicting means, when the gaze point estimated by the gaze point estimating means has moved, re-estimates the information required by the user based on the moved gaze point. 2. The information display device according to claim 1, further comprising: means and a display changing means for instructing the information displaying means to display the information re-predicted by the prediction changing means. 3. The predicting means, when the gaze point estimated by the gaze point estimating means is in the information display means, obtains necessary information of the user based on the gazed information. Predict, if the gaze point is other than the information display means,
The information display device according to claim 1, wherein the information required by the user is predicted based on the part that is being watched. 4. An information display method for displaying information using an information display means, which comprises a gaze point estimating step of estimating a gaze point of a user who uses the information displaying means, and an estimation by the gaze point estimating step. Information comprising a prediction step of predicting information required by the user from the gazed point, and an information display step of displaying the information predicted by the prediction step by the information display means. Display method. 5. The prediction change, when the gaze point estimated by the gaze point estimating step moves, re-predicts necessary information of the user based on the moved gaze point. The information display method according to claim 4, further comprising: a step, and a display change step for instructing the information display means to display the information re-predicted by the prediction change step. 6. In the predicting step, the gaze point estimated in the gaze point estimating step is
When in the information display means, predict the necessary information of the user based on the information that is gazing, if the gaze point is other than the information display means,
The information display device according to claim 4, wherein the information required by the user is predicted based on the part that is being watched.